sglavoie.com – Category: Automation

Using Google Sheets as a database to extract data converted to Markdown

Sébastien Lavoie — Sun, 19 Dec 2021 21:00:00 -0600

Introduction

The purpose of this tool is to parse and convert the content of a Google Sheets spreadsheet to Markdown. In this specific case, I just wanted a simple script to reduce the friction in getting a valid output I could use to maintain a “learning log” on my website. So that gives me a quick way to input data in a spreadsheet and then I can take the time whenever needed to update the learning progress as desired by reformatting any of the entries and optionally adding notes to them.

General software requirements

This tool should:

Be able to read a private spreadsheet with a service account configured with the Google Sheets API.
Convert rows from a Google Sheets spreadsheet into Markdown (see Input and Output sections below for details).
Skip rendering rows that are missing a value for the Title or Date columns (it couldn’t be rendered in the correct place properly without this information).
- Other columns are optional and the corresponding level of nesting will be skipped if no value was provided (e.g. a category Articles may have an item with no sub-category and could be missing any of the other values except for Title and Date).
Separate entries by month by parsing the Date column.
- A command-line parameter should exist to specify a year from which the date should be extracted.
- The output should be in reverse chronological order (i.e. for a full year with entries for each month, December will appear at the top and January at the bottom).
Output a hierarchical format like the following: Category > Sub-category > Title > Activity & notes
Group entries by category and sub-category (e.g. if a category is named Articles and there is a sub-category named Python, then Python must be nested under Articles for the corresponding month).
Sort on the Date column before doing any parsing on other columns (rows in the spreadsheet can be in non-chronological order).
- Dates from a single year will be kept (either the year received as a parameter or the most recent year found).
Sort Category, Sub-category and Title alphabetically (ascending order from top to bottom).
Not sort the Activity column to preserve the order in which rows were added to the spreadsheet.
Render links only if the Link column is a valid URL.
Not validate the content of columns except for Date and Link (i.e. render the other columns as is).
Display items with no category above those having a category. The same logic would be true for sub-categories.
Format the output:
- Date should become headers (##).
- Category should be bolded.
- Sub-category should be emphasized.
- Title should become a link if the Link column contains a valid URL.
- Activity should be nested under Title, occupying a new line for each activity.
- Notes should appear (italicized and inside parentheses) next to the activity.
  - If there is a value for Notes but no value for Activity, the output would be next to the Title.

Input

A spreadsheet with the following columns (starting with Date):

[additional info]	Date	Category	Sub-category	Title	Activity	Link	Notes
November	13/11/2021	Articles	Python	Python slots, slots, and object layout		https://www.youtube.com/watch?v=Iwf17zsDAnY	understood that X does Y
November	14/11/2021	Books	Software engineering	The Pragmatic Programmer	ch. 1	https://pragprog.com/titles/tpp20/the-pragmatic-programmer-20th-anniversary-edition/	note 1 here
November	15/11/2021	Books	Software engineering	The Pragmatic Programmer	ch. 2		note 2 here too
November	16/11/2021	Books	Software engineering	Clean Code
December	1/12/2021	Documentation	Python	The Python Tutorial	Sections 1-4	https://docs.python.org/3/tutorial/index.html
December	2/12/2021	Books	Software engineering	Clean Code	ch. 2-10
Render available values	2/12/2021	Books		Clean Coder			book note
Render available values	2/12/2021			Python/C API Reference Manual		https://docs.python.org/3/c-api/index.html
Title missing: skip rendering	3/12/2021	Articles
Title missing: skip rendering	3/12/2021	Articles	Software engineering		ch. 3	https://docs.python.org/3/c-api/index.html	note
Date missing: skip rendering		Articles	Python	Python slots, slots, and object layout	Section 2	https://www.youtube.com/watch?v=Iwf17zsDAnY	note not rendered

Output

A Markdown output to the terminal in the following format:

 markdown
 
 
 
## December

- [Python/C API Reference Manual](https://docs.python.org/3/c-api/index.html)
- **Books**
 - Clean Coder (_book note_)
 - _Software engineering_
 - Clean Code
 - ch. 2-10
- **Documentation**
 - _Python_
 - [The Python Tutorial](https://docs.python.org/3/tutorial/index.html)
 - Sections 1-4

## November

- **Articles**
 - _Python_
 - [Python slots, slots, and object layout](https://www.youtube.com/watch?v=Iwf17zsDAnY) (_understood that X does Y_)
- **Books**
 - _Software engineering_
 - Clean Code
 - [The Pragmatic Programmer](https://pragprog.com/titles/tpp20/the-pragmatic-programmer-20th-anniversary-edition/)
 - ch. 1 (_note 1 here_)
 - ch. 2 (_note 2 here too_)

Input/Output example when using this tool.

Installing this tool

Tested only under Python 3.9.7, requires at least Python 3.7+.

Using `pip`

[recommended] Activate a virtual environment.
To install dependencies, run from this directory: pip install -r requirements.txt
Then execute the script to run the tool: python3 get_learning_logs

Using `pipenv`

Run from this directory: pipenv install.
Activate the newly created virtual environment with pipenv shell.
Then execute the script to run the tool: python3 get_learning_logs

How to use this tool

You will need to create an environment file (default path: ~/.learning-logs or edit the path for LEARNING_LOGS_ENV_PATH in get_learning_logs.py). It should look as follows:

 env
 
SERVICE_ACCOUNT_FILEPATH="/path/to/service_account_learning_logs.json"
SPREADSHEET_ID=SPREADSHEET_ID # Found in the URL of the document
WORKSHEET_ID=0 # First sheet is 0 by default, it comes after the URL parameter `gid`

To set up this project, you will have to:

Enable the Google Sheets API for your project (https://support.google.com/googleapi/answer/6158841?hl=en).
In the project, search for “service account” in the search bar and create new credentials.
Download the credentials and put them at the path SERVICE_ACCOUNT_FILEPATH referenced in the environment file.
Create a spreadsheet with the following columns (not necessarily in that order): Date, Category, Sub-category, Title, Activity, Link, Notes.
- The column Date Should match a date format: select all cells from that column, go to Format > Number > Date to apply the expected format (mm/dd/yyyy). You can then input a date (e.g. 11/1/2021) and you will see a calendar pop up when double-clicking on it. Dragging it down to a new cell will create a new date for the next day (in this example, 11/2/2021).
Get its SPREADSHEET_ID (https://docs.google.com/spreadsheets/d/SPREADSHEET_ID_HERE/edit#gid=0) to put it in the environment file.
Get the WORKSHEET_ID (the integer after gid= in the URL of the document).
From the spreadsheet document (or from the folder view in Google Drive), share with the email associated with the service account (e.g. learning-logs-something@project-name-12345.iam.gserviceaccount.com). Read access (“Viewer” permission) is enough.

By now, the script is ready to be called. I just created an alias to execute it more easily by putting the following in ~/.bash_aliases (you will have to adapt the paths of course):

 bash
 
alias learning-logs='~/.local/share/virtualenvs/learning-logs-to-markdown-XJLvhmzn/bin/python3.9 \
 ~/dev/sglavoie/dev-helpers/learning-logs-to-markdown/get_learning_logs.py'

This will output the converted data from the spreadsheet to Markdown as shown above.

Conclusion

This little tool scratched an itch and will be helpful in making the process of updating the learning progress of this website more straightforward, more convenient and less hands-on! You can find its source code on GitHub.

Convenient and lightning fast backups with rsync

Sébastien Lavoie — Sat, 31 Jul 2021 09:01:00 -0500

Introduction

Coming into its fourth year of existence and usage in production environments, this simple script has proven to be a fantastic ally. Despite the plethora of great options out there that have more bells and whistles, I have yet to find a single complaint about this solution as it hinges on the minimalist side, requiring only a working Python 3 installation along with the powerful rsync command-line tool.

It’s one of those pieces of code that perform some really basic tasks that can be scheduled with cron in the background, that one forgets about until a real need to access a backup comes up. This script is barely a Python wrapper calling rsync but still, I felt like sharing it for those who like uncomplicated setups. I’ve literally left this script alone for years some time after the release of Python 3.6 which was released back in December 2016 and it has been going strong since then mostly due to the fact it does not do much more than backing files up and leveraging rsync.

Prerequisites

As briefly mentioned above, this tool uses Python 3.6+ and rsync. That’s it, really.

Setting up the script

This script is being made available on GitHub within a set of other small tools. To get the source code for this particular tool, you can either clone the whole repository, get the content of the files separately or simply copy and paste the current version from below. You will need a settings.json file as well as the Python script that will run the necessary commands.

The configuration file

Right off the bat, you will most probably want to update the array of strings for the data_sources key as well as the string for the key data_destination in the following JSON file.

 json
 
 
 
{
 "data_sources": [
 "/home/sglavoie"
 ],
 "data_destination": "/media/sglavoie/Elements",
 "terminal_width": 40,
 "sep": "=-",
 "log_name": ".backup_log_",
 "log_format": "%y%m%d_%H_%M_%S",
 "rsync_options": [
 "-vaAHh",
 "--delete",
 "--ignore-errors",
 "--force",
 "--prune-empty-dirs",
 "--delete-excluded"
 ],
 "backup_exclude": ".backup_exclude"
}

By default, the following options are passed to rsync:

Option	Description
`-vaAH`	verbose, archive, ACLs, hard-links (preserve)
`--delete`	“delete extraneous files from destination dirs”
`--ignore-errors`	“delete even if there are I/O errors”
`--force`	“force deletion of directories even if not empty”
`--prune-empty-dirs`	“prune empty directory chains from the file-list”
`--delete-excluded`	“also delete excluded files from destination dirs”

Description of available settings

Name of setting	Description
`data_sources`	Directories to backup, supplied as a list of strings (no slash at the end).
`data_destination`	Single destination of the files to backup, supplied as a string. This can be overridden when passing option `-d` or `--dest` to the script
`terminal_width`	Line length in the terminal, used for printing separators.
`sep`	Separator to use along with `terminal_width`.
`log_name`	Sets the prefix of the log filename.
`log_format`	This goes right after `log_name` as a suffix.
`rsync_options`	Options to use with rsync as a list of strings.
`backup_exclude`	Default file in each source in `data_sources` where files/directories will be ignored.

The script

 python
 
 
 
"""
Script that uses `rsync` to make a simple and convenient backup.
Note: requires Python 3.6+. No other Python third-party libraries required.
"""
import argparse
import datetime
import glob
import json
import os
import pathlib
import subprocess


def run_backup():
 """This is where all the action happens!"""
 settings = get_settings()

 # initiate the parser
 parser = argparse.ArgumentParser("backup")
 parser.add_argument(
 "-c",
 "--clear",
 help="Delete all log files for current source in DATA_SOURCES.",
 action="store_true",
 )
 parser.add_argument(
 "-s",
 "--src",
 dest="source",
 default=None,
 help="Specify an alternative source to backup as a string.",
 action="store",
 )
 parser.add_argument(
 "-d",
 "--dest",
 dest="destination",
 default=None,
 help="Specify an alternative destination for backup as a string.",
 action="store",
 )

 # read arguments from the command line
 arguments = parser.parse_args()

 # check for --source or -s
 # Replace potential list of sources to backup with this one
 if arguments.source is not None:
 if os.path.isdir(arguments.source):
 settings["data_sources"] = [arguments.source]
 else:
 print("Please enter a valid source to backup.")
 exit(0)

 # check for --clear or -c
 if arguments.clear:
 clear_logs(
 data_sources=settings["data_sources"],
 log_name=settings["log_name"],
 )

 # check for --dest or -d
 # Replace destination to backup with this one
 if arguments.destination is not None:
 if os.path.isdir(arguments.destination):
 settings["data_destination"] = arguments.destination
 else:
 print("Please enter a valid destination.")
 exit(0)

 # don't run the script if the destination doesn't exist
 if not os.path.isdir(settings["data_destination"]):
 print(
 f"The destination doesn't exist.\n({settings['data_destination']})"
 )
 exit(0)

 backup_all_sources(settings)


def backup_all_sources(settings: dict) -> None:
 """Iterate over all sources to backup."""
 for source in settings["data_sources"]:
 date_now = datetime.datetime.now()
 log_format = datetime.datetime.strftime(
 date_now, settings["log_format"]
 )
 log_filename = f"{source}/{settings['log_name']}{log_format}"
 log_option = f"--log-file={log_filename}"

 backup_source = settings["backup_cmd"].copy()
 backup_source.extend([log_option])

 # files to ignore in backup
 exclude_file = f"{source}/{settings['backup_exclude']}"

 if os.path.exists(exclude_file):
 exclude_option = f"--exclude-from={exclude_file}"
 backup_source.extend(
 [exclude_option, source, settings["data_destination"]]
 )
 else:
 # skips '--exclude-from' option if no file is found
 backup_source.extend([source, settings["data_destination"]])

 settings["source"] = source
 settings["backup_source"] = backup_source
 settings["log_filename"] = log_filename

 try:
 backing_source(settings)
 except KeyboardInterrupt:
 print("\nKeyboardInterrupt: Exiting operations.")
 exit(0)


def backing_source(settings: dict) -> None:
 """Print information to STDOUT and to `log_filename` and executes the
 rsync command."""
 print(settings["sep"] * settings["terminal_width"])

 cmd_executed = " ".join(settings["backup_source"])
 msg_executed = f"Command executed:\n{cmd_executed}\n"
 print(msg_executed)

 with open(settings["log_filename"], mode="w") as log_file:
 log_file.write(f"{msg_executed}\n")

 try:
 child = subprocess.Popen(settings["backup_source"])
 _ = child.communicate()[0] # call communicate to get the return code
 rc = child.returncode
 except FileNotFoundError:
 print(f"FileNotFoundError: Is the `rsync` tool installed?")
 exit(1)

 print(f"\nBackup completed for: {settings['source']} (return code: {rc})")
 print(settings["sep"] * settings["terminal_width"])


def clear_logs(data_sources: list, log_name: str) -> None:
 """Clears log files for each source specified in SETTINGS."""
 for source in data_sources:
 # Retrieve a list of all matching log files in `source`
 log_files = glob.glob(f"{source}/{log_name}*")
 if log_files == []:
 print(f"\nThere is no log file to delete in {source}.")
 exit(0)
 else:
 print(f"Log files in {source}:")
 for log_file in log_files:
 print(log_file)
 if user_says_yes():
 for log_file in log_files:
 os.remove(log_file)
 print("Log files deleted.")
 print("Exiting script...")
 exit(0)


def user_says_yes(
 msg: str = "\nDo you want to delete log files for this source? (y/n) ",
) -> bool:
 """Asks the user to enter either "y" or "n" to confirm. Returns boolean."""
 choice = None
 while choice is None:
 user_input = input(msg)
 if user_input.lower() == "y":
 choice = True
 elif user_input.lower() == "n":
 choice = False
 else:
 print('Please enter either "y" or "n".')
 return choice


def get_settings() -> dict:
 """
 Get the settings from `settings.json`.

 Returns:
 dict: Containing all settings used by the tool.
 """
 directory = pathlib.Path(__file__).parent.absolute()
 with open(directory / "settings.json") as fp:
 settings = json.load(fp)

 backup_cmd = ["rsync"]
 backup_cmd.extend(settings["rsync_options"])
 settings["backup_cmd"] = backup_cmd

 return settings


if __name__ == "__main__":
 run_backup()

How to use

One-time setup

Set all values in settings.json to suit your needs.
Make sure that the backup destination is available/mounted. A simple warning will be echoed if the destination can’t be found.
Call the Python script. As an example, if it’s located at /home/user/backup/rsync_backup.py, then you could put the following alias in ~/.bash_aliases (or equivalent)*:

 bash
 
alias backup='python3 /home/user/backup/rsync_backup.py'

* Python may be called differently on your system, e.g. simply python instead of python3.

In this instance, the script can now be executed in a terminal with the keyword backup along with optional arguments.

Using as a daily driver

Make sure the destination to back files up is available/mounted and simply call the script (e.g. backup in this example).

The output will look something like this:

 bash
 
 
 
$ backup
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Command executed:
rsync -vaAHh --delete --ignore-errors --force --prune-empty-dirs \
 --delete-excluded --log-file=/home/sglavoie/.backup_log_210731_09_28_09 \
 --exclude-from=/home/sglavoie/.backup_exclude /home/sglavoie /tmp

building file list ... done
deleting sglavoie/.backup_log_210731_08_47_24
sglavoie/
sglavoie/.bash_aliases
...

Avoid deleting files at the destination

The rsync command first builds a list of the files that are part of the backup and will go on adding new files, updating existing files as well as removing files that are found in the destination but not in the source anymore according to the default list of flags passed to it in this implementation. It is therefore important to note that if you would like to back up files in the destination that no longer exist in the source, a convenient solution is simply to create a new folder at the root level inside the destination directory and manage those files separately.

For instance, I sometimes want to back up large video files but there is rarely a need to update them. In this case, it would be possible to add the directories to exclude in the file .backup_exclude (default configuration option) like so:

Videos/
Dropbox/other/folder/

Then, these directories will be ignored when backing up the source (if they are nested inside the source directory to start with!). If the destination is set to /media/user/backup and the source set to /home/user/my_folder, then the script would recreate the “root” directory of the source if it doesn’t exist on the destination:

bash

$ ls /media/user/backup
my_folder

You could keep track of those heavy files that do not need to be backed up frequently by putting them manually next to my_folder in /media/user/backup. This will lead to huge speed ups when running the script if you track many files in this way!

Conclusion

This was a quick overview of what rsync can offer: I neither touched upon its remote syncing capabilities nor the vast majority of its options, for which you can find a lot more information about by typing man rsync to display the manual page in the terminal.

Without making use of any fancy features of recent versions of Python or rsync, I have found this simple backup procedure to have worked flawlessly for a long time. Hardware inevitably fails at some point and after having heard about countless examples of people losing some or all of their most precious files (including my own story…), I have come to enjoy backing my system with this little script.

I wouldn’t trust my external drive to last forever either, hence I also rely on Dropbox and Google Drive for documents I find are a good fit. But overall, I just do not tend to back up every single system file in the cloud, so there is still a convincing use case where daily and complete backups become a possibility when using a fast approach like rsync to incrementally save snapshots.

More resources and references

rsync official website – samba.org.
Keeping Linux files and directories in sync with rsync – RedHat, going through the main options offered by the tool.
How To Use Rsync to Sync Local and Remote Directories – DigitalOcean, demoing how to use the tool and covering the case of remote file access.

Exponentiation in Modular Arithmetic Made Easy

Sébastien Lavoie — Sat, 27 Apr 2019 19:52:00 -0500

Introduction

The following Python script will by no means provide any useful idea to compete against the RSA algorithm , but it does give you an idea of how a simple technique about reducing the size of an exponent in modular arithmetic can bring you closer to using much larger numbers than the ones you could normally use in the Python interpreter.

The script in action

 python
 
 
 
"""Compute the result of a^b (mod k) by using the exponentiation technique.
The goal here is not efficiency, even though the program is actually pretty
fast: the algorithm is applied manually for demonstration purposes.
Testing on a modest Intel Core i5, having `a` and `b` each set to a random
number containing 2,000 digits and `k` set to a modulo of a number containing
20 digits, results are printed in about 3.3 seconds."""

def binary_remainders(num_b):
 """Take `b` and return the binary equivalent in a list of remainders."""
 remainders = []
 quotient = num_b
 while True:
 prev_quotient = quotient
 quotient //= 2
 remainder = prev_quotient % 2
 remainders.append(remainder)

 if quotient == 0:
 break

 return remainders


def powers_of_two(remainders=None):
 """Return a list of the value of powers of two that form the
 exponent`b`."""
 if remainders is None:
 return None

 powers = []
 for index, remainder in enumerate(remainders):
 if remainder == 1:
 powers.append(2**index)

 return powers


def compute_intermediate_congruences(num_a, num_k, powers=None):
 """Compute all necessary intermediate results of congruence in `mod k` for
 powers of 2 in `powers` to form the number `b`."""
 if powers is None:
 return None

 go_up_to = max(powers)

 intermediate_results = {1: num_a} # Build dictionary to store all results
 start_value = 2 # First power of two to calculate congruence
 congruence = num_a
 while start_value <= go_up_to:
 # value to use for next power of 2
 congruence = congruence ** 2 % num_k
 intermediate_results[start_value] = congruence
 start_value *= 2

 return intermediate_results


def compute_final_congruence(num_k, powers=None, intermediate_results=None):
 """Take all relevant values from `intermediate_results` matching powers in
 `powers`, multiply them together and calculate this number `mod k` to get
 the final result."""
 if intermediate_results is None or powers is None:
 return None

 # store all required values from `intermediate_results`
 congruent_results = []
 for power in powers:
 for key, value in intermediate_results.items():
 if key == power:
 congruent_results.append(value)

 total = 1
 for result in congruent_results:
 total = result * total # Multiply all results together

 return total % num_k # final congruence we are looking for


def compute_congruence(num_a, num_b, num_k):
 """Return `c`, the result of `a^b (mod k)`."""
 num_a = num_a % num_k # Make sure `a` is smaller than `k`

 # Reduce `b` to list of remainders in binary
 remainders = binary_remainders(num_b)

 # Build a list of the powers of 2 forming `b`
 powers = powers_of_two(remainders)

 # Build a list of necessary intermediate results to reach
 # the value of `b` from powers of 2: finds congruence for
 # smaller powers of 2 and store them in a list.
 intermediate_results = compute_intermediate_congruences(num_a, num_k,
 powers)

 # Multiply all relevant intermediate results `mod k` to get the final
 # congruence of `a^b (mod k)`.
 return compute_final_congruence(num_k, powers, intermediate_results)


if __name__ == '__main__':
 print("We will calculate a^b (mod k). Enter only integers.")
 NUM_A = int(input("Provide `a`: "))
 NUM_B = int(input("Provide `b`: "))
 NUM_K = int(input("Provide `k`: "))

 FINAL_RESULT = compute_congruence(NUM_A, NUM_B, NUM_K)
 print(FINAL_RESULT)

Conclusion

It was very interesting to see how a technique that’s applied manually will work wonders with such large numbers. The Python interpreter can barely calculate numbers with exponents with seven digits or more, while a basic approach like the one shown above can quickly churn out the results for seemingly quite large numbers.

As always, this code is available on GitHub.

Have You Been Pwned?

Sébastien Lavoie — Thu, 21 Mar 2019 13:42:00 -0600

Introduction

Have I Been Pwned? has a very simple and accessible API that’s perfect to work with as a beginner. There are different ways to retrieve information and the script below is only one glimpse into the numerous possibilities available. It is built in a way that makes it easy to add features and functionality thanks to Click, a wonderful “command line interface toolkit”.

Python to the rescue

The following script uses Click to build a tiny command-line interface instead of more traditional tools like argparse. You will also need requests to peek inside the API. Both can be obtained as follow in a terminal:

bash

$ pip install click requests

And now, the pièce de résistance.

 python
 
 
 
"""
Simple script that will take advantage of haveibeenpwned.com's API to
find out if your password has been breached.
"""
# Standard library imports
import hashlib

# Third party library imports
import click
import requests


# Needed to access API
PASS_URL = 'https://api.pwnedpasswords.com/range/'

# Script information
VERSION = '0.1.1'
LOGO = r'''
 __ __ __
| |--.---.-.--.--.-----. |__| | |--.-----.-----.-----.
| | _ | | | -__| | | | _ | -__| -__| |
|__|__|___._|\___/|_____| |__| |_____|_____|_____|__|__|

 __ _____
.-----.--.--.--.-----.-----.--| ||__ |
| _ | | | | | -__| _ |', ,-'
| __|________|__|__|_____|_____| |--|
|__| '--' '''


def print_info(context, param, value): # `param` required by `click`
 """Print information about the program and exit."""
 if not value or context.resilient_parsing:
 return
 click.secho(f'{LOGO}\n{"Python 3.6+ Checker".rjust(60)}\n'
 f'{("v" + VERSION).rjust(60)}', fg='blue', bold=True)
 context.exit()


def pass_to_sha1(password):
 """Will convert `password` with SHA-1 algorithm and return the short
 version with the first five characters and the long version with
 everything but the first five characters as a tuple."""

 my_password = password.encode() # converts to byte string

 complete_hash = hashlib.sha1(my_password).hexdigest().upper()
 long_hash = complete_hash[5:]
 short_hash = complete_hash[:5]

 return short_hash, long_hash


def check_password(short_hash, long_hash):
 """Consult haveibeenpwned.com to see how many times the password has
 been breached."""
 click.secho('Please note that only the 5 first characters from your '
 'SHA-1 ENCRYPTED password are sent to haveibeenpwned.com.',
 fg='yellow', bold=True)
 response = requests.get(PASS_URL + short_hash)
 lines = response.text.split()
 for line in lines:
 if long_hash in line:
 num_times = 'occurrences' if int(line[36:]) > 1 else 'occurence'
 click.secho('Your password was found!', fg='red', bold=True)
 click.echo(f'→ {line[36:]} {num_times}')
 break
 else:
 click.secho('Your password was NOT found!', fg='green')


@click.group()
@click.option(
 '--info',
 is_flag=True,
 callback=print_info,
 expose_value=False,
 is_eager=True,
 help="Print some information about the program and exit.")
@click.version_option(version=VERSION, message="%(version)s")
def hibp():
 """A simple command-line interface to make use of
 haveibeenpwned.com's data.

 Type `python3 pwned.py usage` for more info (or whatever way you
 call Python 3)."""


@hibp.command()
def usage():
 """Give examples on how to use this script."""
 click.secho('Examples:', fg='blue')
 click.secho('python3 pwned.py check --help for more info.', fg='green')
 click.secho('python3 pwned.py check -p MyPasswordHere', fg='yellow')
 click.secho('python3 pwned.py check --password MyPasswordHere', fg='green')

 click.echo('')
 click.secho('For more complicated passwords, you have to use quotes and ',
 fg='blue')
 click.secho('escape symbols with \\ where appropriate:', fg='blue')
 click.secho('python3 pwned.py check -p "as0d9\\"asg0\'\'A=)SYD"',
 fg='green')


@hibp.command()
@click.option(
 '-p',
 '--password',
 default=None,
 help="Reveal if match exists with `password`.")
def check(password):
 """Tell if your password has been breached."""
 if password:
 short_hash, long_hash = pass_to_sha1(password)
 check_password(short_hash, long_hash)
 if not password:
 click.secho('Run `pwned.py check --help` for more info.', fg='red')


if __name__ == '__main__':
 hibp()

Example usage

Conclusion

If you are familiar with Python, you can easily make sense of how this little tool works by reading its code. You can also have a look at this repository on GitHub if you want to fork it or reuse it any other way you wish. I have personally found it much easier to work with Click instead of argparse and optparse. I haven’t got to try docopt yet, but it is almost equally as popular as Click on GitHub at the time of this writing and its Pythonic way makes it another great option to consider.

Number of Paths in a Grid… or in Life

Sébastien Lavoie — Thu, 14 Mar 2019 20:38:00 -0600

Introduction

How would you go about determining the number of possible paths in a grid of size n if your initial position is at the top left corner and your destination is the bottom right corner, knowing you can only go right or down at each move? Let’s look at a grid of size 3 × 3:

Even at such a small size, one needs some concentration. Sure, it can be done on paper, but there must be a better way…

Python to the rescue

Once again, programming comes in handy! Life is complicated, but there are aspects of it that can be solved more easily than previously thought when we use the right tools. Knowing that this particular situation corresponds to values from Pascal’s Triangle might help, but it sure is more fun to come up with a different solution. Here is how it occurred to me in Python:

 python
 
 
 
'''
This approach consists in getting the number of possible paths for each
position where we stop in a grid. We know that once we reach the bottom
or the right of the grid, there is only one path possible from there,
so we take advantage of this fact and calculate the total number of
possible paths starting from the bottom right corner. A 2x2 grid would
have 9 indices starting at 0:
012
345
678

We start at index 8 and will calculate the number of possible paths for
all previous positions. From index 8, we calculate that index 4 must
have 2 possible paths to reach index 8 (4>7>8 and 4>5>8). We then find
that index 3 has 3 possible paths since from there we can go right to
index 4 (2 paths) or down to index 6 (1 path). We continue by checking
index 1, which would be the sum of the number of paths at index 4 and
index 2, which is 3 possible paths from there. Now, we are left at the
beginning and the total number of possible paths is index 3 + index 1
(3 + 3 = 6).

The values used would then be tweaked a little depending on the size
of the grid, but the algorithm remains the same. Here is how it works
concretely:

- Get the number of positions in the grid. 2x2 means 9 positions by
counting all intersections. With a 2x2 starting at index 0, we have the
following positions:
012
345
678

- Generate a list corresponding to the length of previously found number
 of positions and give a value of 1 to each position. For a 2x2, it is
 [1,1,1,1,1,1,1,1,1]
- Determine the multiple to be used to check if an index falls in the
 first column (grid_size + 1). The indices in the first column are 0, 3
 and 6, so multiple = 3.
- Start from the bottom right corner of the grid (index 8). Get the
 index of that position starting at zero (length of positions - 1).
- Start a while loop:
 - Check if current position is equal to grid size, meaning we got to
 the last position to evaluate. Return the value of the first index if
 this is the case.
 - Check if current position is a multiple of 3 in that example.
 If so, we skip and go back one index. For instance, if index was 6
 (multiple of 3), go back to index 5.
 - Determine the index of position to the left of current position
 and the index of position above current position. Sum the two together
 and give that value to the index that corresponds to one step left
 and one step up in the grid. Starting at 8, we sum index 7 and index
 5 (result = 2) and set that value for index 4, which is the number of
 possible paths from that index number.
 - Decrement current position by one and repeat the process until we
 break out of the loop.
'''

GRID_SIZE = 100


def num_paths(grid_size):
 '''Return the number of possible paths in a grid of size `grid_size`
 when going only right and down as an integer.'''
 num_points = (grid_size + 1) ** 2
 path_list = [1] * num_points
 multiple = grid_size + 1 # Indices of first column match this multiple

 # Last index in list
 starting_point = len(path_list) - 1

 while True:
 if starting_point == grid_size:
 return path_list[0]
 if starting_point % multiple == 0:
 starting_point -= 1
 else:
 previous = starting_point - 1
 previous2 = starting_point - multiple
 left_up_index = previous2 - 1
 left_up_value = sum([path_list[previous], path_list[previous2]])
 path_list[left_up_index] = left_up_value
 starting_point -= 1


if __name__ == '__main__':
 print(num_paths(GRID_SIZE))

In the end, the idea was to find the number of possible paths that correspond to each possible move, starting from the destination and reverse engineering the number of possibilities until we get to the starting point.

Conclusion

As it turns out, this strategy of reverse engineering things works pretty well when applied to real-life examples. When setting a goal, for instance loosing 10 kilograms in one year, it can sound a lot less daunting to find out what this represents by month (0.83 kg) or even by week (0.19 kg) instead of fixating on the big gap from your current situation. For short-term goals, this can be reduced to daily or even hourly outcomes.

This goes hand in hand with the advice found in Atomic Habits, an excellent book written by James Clear:

A 1% daily improvement leads to a version of yourself that’s 37 times better in one year.

If that’s not motivating enough to start taking little steps with consistency towards your dreams, I don’t know what is.

Binary to Decimal, Decimal to Binary Converter

Sébastien Lavoie — Mon, 18 Feb 2019 21:30:00 -0600

Introduction

There is nothing new here in terms of algorithms, but it was interesting to figure out how to apply those concepts with Python. This works for integer numbers, but can be easily adapted to convert fractions.

Binary to Decimal

 python
 
 
 
"""
Converts a binary number to a decimal number.
"""


def bin_to_dec(number):
 """Algorithm that converts a binary number to a decimal number.

 Receives and returns a string."""

 # get the index of power of first digit
 # e.g. 1001 → first index power is 3 (2^3)
 first_power = len(number) - 1

 total = 0 # initialize variable to store final result

 # iterate over each number and if it is '1', calculate the value
 # based on its position and add it to the total
 for index, char in enumerate(number):
 if char == '1':
 total += 2 ** (first_power - index)

 # give back the answer to the script
 return total


if __name__ == '__main__':
 # retrieve binary number from input as string to parse
 BIN_NUM = input()

 # pass binary number to algorithm and store the result
 RESULT = bin_to_dec(BIN_NUM)

 # print the solution to the console
 print(RESULT)

Decimal to Binary

 python
 
 
 
"""
Converts a decimal number to a binary number.
"""


def dec_to_bin(number):
 """Algorithm that converts a decimal number to a binary number.

 Receives an integer and returns a string."""

 if number == 0: # if number is 'zero', the answer is 'zero'
 return 0

 remainders = [] # list that stores the remainders

 # iterate over the initial given decimal and divide it by two
 # until it gets to 'zero'. At each step, add the remainder in
 # the above list
 while number > 0:
 remainders.append(number % 2)
 number = number // 2

 # reverse the list of remainders, as the first remainder is the
 # right most digit in the answer
 remainders.reverse()

 # concatenate all the digits from the list of remainders from left
 # to right to display the final binary number
 answer = ''
 for digit in remainders:
 answer += str(digit)

 # give the answer back to the script
 return answer


if __name__ == '__main__':
 # retrieve decimal number from input as integer
 DEC_NUM = int(input())

 # pass decimal number to algorithm and store the result
 RESULT = dec_to_bin(DEC_NUM)

 # print the solution to the console
 print(RESULT)

Conclusion

This was a very quick programming session that’s been useful to make new concepts stick. The best learning strategies are to test yourself constantly in any way possible and to teach clearly what you know, which is what I tried to do in the end. By the way, I found the course Learning How to Learn on Coursera to be quite satisfying and I would invite you to have a look if that’s a topic you’re interested in!

Bash History Cleaner

Sébastien Lavoie — Sun, 23 Dec 2018 19:35:00 -0600

Introduction

This is a Python 3.6+ script that helps to clean the file containing the Bash history commands. It will remove any line matching a specified regular expression and can also remove any line starting with an alias.

The idea behind this small utility was simple:

The Bash history file (usually located in ~/.bash_history) contains much of the work one ends up doing in the terminal.
The history can grow large over time and it becomes more cumbersome to find interesting information in all that clutter, such as a rarely used command with specific flags.
By removing all superfluous commands that are repeated often and which give no real benefit in certain contexts (such as ls, cd, cat, etc.), the history is much cleaner and easier to navigate and actually becomes much more useful in my opinion.
True, it will be harder to follow the bread crumbs for everything you did, but I haven’t come across a situation where having access to yet another empty ls or cd has proven necessary and reading .bash_history doesn’t make for a great narrative story either.

Make history in a big way

I took advantage of the fact that the history can be cleaned with the script you are about to see and set up what is known as an eternal history which, as it sounds like, can grow infinitely big! All you have to do is append the following lines to the file ~/.bashrc:

 bash
 
 
 
# Eternal bash history.
# ---------------------
# Undocumented feature which sets the size to "unlimited".
# http://stackoverflow.com/questions/9457233/unlimited-bash-history
export HISTFILESIZE=-1
export HISTSIZE=-1
export HISTTIMEFORMAT="[%F %T] "
# Change the file location because certain bash sessions truncate
# .bash_history file upon close.
# http://superuser.com/questions/575479/bash-history-truncated-to-500-lines-on-each-login
export HISTFILE=~/.bash_eternal_history
# Force prompt to write history after every command.
# http://superuser.com/questions/20900/bash-history-loss
PROMPT_COMMAND="history -a; $PROMPT_COMMAND"

Bash History Cleaner

And here is the script in question *. It comes in two files that need to be in the same directory:

One is a Python file that needs to be launched from the terminal with Python 3.
The other file, settings.json, is a JSON file used to store the settings of the script, which will be detailed below.

* _{Improvements to the original script can be found on Github. To keep this
article a bit more readable, the original version is shown.}

`bash_history_cleaner.py`

 python
 
 
 
'''
Python script that helps to clean the file containing the Bash history
commands.

Note: Requires Python 3.6+

It will remove any line matching a specified regular expression and can
also remove any line starting with an alias.

Description of available settings in `settings.json`:

 "home_directory": Absolute path to user's home directory.

 "history_file": Name of file where the history will be cleaned up.

 "aliases_file": Name of file where Bash aliases are set up.

 "ignore_patterns": List of patterns to ignore in `history_file`.
 Each line where a pattern is found will be deleted.
 → Patterns are specified as regular expressions.

 "add_aliases": Boolean. If set to `true`, aliases from `aliases_file`
 will be added to `ignore_patterns`.

 "aliases_match_greedily":
 Boolean. If set to `true`, any line in `history_file`
 starting with an alias in `aliases_file` will be
 deleted. If set to `false`, delete line if the alias is
 the content of the whole line (with optional space at
 the end): `false` matches "^alias$" or "^alias $" only.

 "backup_history": Boolean. If set to `true`, `history_file` will be backed
 up in the same directory with a name ending in .bak
 based on the current date.

 "delete_logs_without_confirming":
 Boolean. If set to `true`, script with flag `-c` will
 automatically delete all the backup files found for
 `history_file`.
'''
from datetime import datetime
from pathlib import Path
import argparse
import fileinput
import glob
import json
import os
import re


def get_current_path():
 '''Returns the current working directory relative to where this script
 is being executed.'''
 return Path(__file__).parents[0]


def user_says_yes(message=""):
 '''Check if user input is either 'y' or 'n'. Returns a boolean.'''

 while True:
 choice = input(message).lower()
 if choice == 'y':
 choice = True
 break
 elif choice == 'n':
 choice = False
 break
 else:
 print("Please enter either 'y' or 'n'.")

 return choice


def delete_logs(settings: dict, history_file: str):
 '''Delete log files in `home_directory` based on `history_file`.'''

 # Retrieve a list of all matching log files
 log_files = glob.glob(f'{history_file}_*.bak')

 if log_files == []:
 print("There is no log file to delete.")
 else:
 print(f'Log files found in {settings["home_directory"]}:')

 for log_file in log_files:
 print(log_file)

 if settings['delete_logs_without_confirming']:
 for log_file in log_files:
 os.remove(log_file)
 print('Log files deleted.')
 return

 message = ("\nDo you want to delete those log files? [y/n] ")

 if user_says_yes(message=message):
 for log_file in log_files:
 os.remove(log_file)
 print('Log files deleted.')
 return

 print('Operation aborted.')
 return


def generate_date_string() -> str:
 '''Return date formatted string to backup a file.'''
 return datetime.strftime(datetime.today(), '_%Y%m%d_%H%M%S.bak')


def load_settings(settings_file: str) -> dict:
 '''Load settings in the script. Return them as a dictionary.'''
 with open(settings_file, "r") as read_file:
 settings = json.load(read_file)
 return settings


def get_list_aliases(bash_aliases_file: str, settings: dict) -> list:
 '''Retrieve the name of all the aliases specified in `bash_aliases_file`.

 Return aliases as a list of strings formatted as regular expressions.'''

 match_whole_line = bool(settings['aliases_match_greedily'])

 with open(bash_aliases_file) as file:
 content = file.read().splitlines() # one alias per line
 aliases_list = []
 for line in content:
 try:
 # Get the actual alias in each line
 # Use negative lookbehind to remove 'alias ' at the beginning.
 # Matches anything after that is a dot, digit, underscore
 # or letter (will stop at the equal sign: alias blah='...')
 alias = re.search(r'(?<!not )alias ([\.\w]*)', line).group(1)

 # If for some reason the alias cannot be extracted, skip it
 # If search doesn't match, it's of type None and won't work
 except AttributeError:
 continue

 if match_whole_line:
 # Match the whole line if it starts with the alias.
 alias = f'^{alias}( )?$|^{alias} .*'
 else:
 # Will match only when alias is the whole content of the line,
 # followed by optional space.
 alias = f'^{alias}( )?$'

 # Escape dots in alias
 alias = alias.translate(str.maketrans({".": r"\."}))

 aliases_list.append(alias)

 return aliases_list


def clean_bash_history(settings: dict, history_file: str):
 '''Modify in place `history_file` by removing every line where
 `ignore_patterns` is found.

 Optionally, add a list of aliases to `ignore_patterns` with
 `aliases` based on the value of `add_aliases` in settings.json.'''

 if settings['backup_history']:
 backup_str = generate_date_string()
 file_input = fileinput.FileInput(history_file,
 inplace=True,
 backup=backup_str)
 else:
 file_input = fileinput.FileInput(history_file,
 inplace=True)

 with file_input as file:
 for line in file:
 has_match = False
 for pattern in settings['ignore_patterns']:
 matched = re.compile(pattern).search
 if matched(line):
 has_match = True
 break
 # If no match is found (nothing to ignore), print the line
 # back into the file. Otherwise, it will be empty.
 if not has_match:
 print(line, end='') # Line already has carriage return


def launch_cleanup(settings: dict, history_file: str, aliases_file: str):
 '''Main function that launches the cleanup process.'''

 bash_aliases = None
 if settings['add_aliases']:
 try:
 bash_aliases = get_list_aliases(aliases_file, settings)

 # add aliases to list of patterns to ignore
 settings['ignore_patterns'].extend(bash_aliases)
 except FileNotFoundError:
 print(f"File not found: {aliases_file}")
 quit()
 try:
 clean_bash_history(settings, history_file)
 except FileNotFoundError:
 print(f"File not found: {history_file}")


if __name__ == '__main__':
 SETTINGS_FILE_PATH = get_current_path() / 'settings.json'
 SETTINGS = load_settings(SETTINGS_FILE_PATH)
 ALIASES_FILE = SETTINGS['home_directory'] + '/' + SETTINGS['aliases_file']
 HISTORY_FILE = SETTINGS['home_directory'] + '/' + SETTINGS['history_file']

 # initiate the parser to check all the arguments passed to the script
 PARSER = argparse.ArgumentParser()
 PARSER.add_argument(
 '-c', '--clear', help='Delete all log files', action='store_true')

 # read arguments from the command line
 ARGUMENTS = PARSER.parse_args()

 if ARGUMENTS.clear:
 delete_logs(SETTINGS, HISTORY_FILE)
 quit()

 launch_cleanup(SETTINGS, HISTORY_FILE, ALIASES_FILE)

`settings.json`

 json
 
 
 
{
 "home_directory": "/home/sglavoie",
 "history_file": ".bash_eternal_history",
 "add_aliases": true,
 "aliases_file": ".bash_aliases",
 "aliases_match_greedily": true,
 "backup_history": true,
 "delete_logs_without_confirming": false,
 "ignore_patterns": [
 "^\\#\\d+",
 "^$",
 "^(\\.\\/)?pip$",
 "^(\\.\\/)?python.*$",
 "^\\.\\.$",
 "^alias",
 "^cd ",
 "^cd$",
 "^cd..$",
 "^fg$",
 "^df( )?",
 "^du( )?",
 "^exit$",
 "^git branch",
 "^git checkout master",
 "^git log$",
 "^git push$",
 "^git status$",
 "^git stauts$",
 "^kill \\d+.*",
 "^ls -a$",
 "^ls$",
 "^make|make install$",
 "^man ",
 "^pelican( )?$",
 "^pip install -r requirements.txt",
 "^pip.* list|pip.* show$",
 "^source ",
 "^sudo apt-get autoclean|sudo apt-get autoremove$",
 "^sudo apt-get dist-upgrade|sudo apt-get update$",
 "^which "
 ]
}

Description of available settings in `settings.json`

Name of setting	Description
`home_directory`	Absolute path to user’s home directory.
`history_file`	Name of file where the history will be cleaned up.
`aliases_file`	Name of file where Bash aliases are set up.
`ignore_patterns`	List of patterns to ignore in `history_file`. Each line where a pattern is found will be deleted. Patterns are specified as regular expressions.
`add_aliases`	Boolean. If set to `true`, aliases from `aliases_file` will be added to `ignore_patterns`. (Default: `true`)
`aliases_match_greedily`	Boolean. If set to `true`, any line in `history_file` starting with an alias in `aliases_file` will be deleted. If set to `false`, delete line if the alias is the content of the whole line (with optional space at the end): `false` matches “^alias$” or “^alias $” only.
`backup_history`	Boolean. If set to `true`, `history_file` will be backed up in the same directory with a name ending in .bak based on the current date. (Default: `true`)
`delete_logs_without_confirming`	Boolean. If set to `true`, script with flag `-c` will automatically delete all the backup files found for `history_file`. (Default: `false`)

Anecdotal evidence of satisfying performances

Performance-wise, this scans ~8,300 lines per second on my modest Intel Core i5 laptop with files of over 200,000 lines long. Not that I type so much stuff in the terminal: I just duplicated many lines.

Conclusion

This is a simple solution to an nonexistent problem, but it was in the end very instructive to me nonetheless. You may even find a use for it! Otherwise, you might use the same functions for other files such as logs! If you would like to take a closer look at the source in a more convenient way, you can find the code available on Github.

sglavoie.com – Category: Automation

Using Google Sheets as a database to extract data converted to Markdown

Introduction

General software requirements

Input

Output

Installing this tool

Using pip

Using pipenv

How to use this tool

Conclusion

Convenient and lightning fast backups with rsync

Introduction

Prerequisites

Setting up the script

The configuration file

Description of available settings

The script

How to use

One-time setup

Using as a daily driver

Avoid deleting files at the destination

Conclusion

More resources and references

Exponentiation in Modular Arithmetic Made Easy

Introduction

The script in action

Conclusion

Have You Been Pwned?

Introduction

Python to the rescue

Example usage

Conclusion

Number of Paths in a Grid… or in Life

Introduction

Python to the rescue

Conclusion

Binary to Decimal, Decimal to Binary Converter

Introduction

Binary to Decimal

Decimal to Binary

Conclusion

Bash History Cleaner

Introduction

Make history in a big way

Bash History Cleaner

bash_history_cleaner.py

settings.json

Description of available settings in settings.json

Anecdotal evidence of satisfying performances

Conclusion

Using `pip`

Using `pipenv`

`bash_history_cleaner.py`

`settings.json`

Description of available settings in `settings.json`