sglavoie.com – Tag: Python

From a plan to unattended agent runs: how I drive coding agents one session at a time

Sébastien Lavoie — Fri, 09 Oct 2026 16:20:00 -0600

Written with AI: This post was drafted with an AI assistant, working from my notes, code and tools, then reviewed and edited by me. The experiences and opinions are mine, and so are any mistakes.

Introduction

Coding agents got good enough, quickly enough, that the bottleneck in my day moved. It’s no longer typing the code; it’s making sure the code that gets typed is the code I wanted, and that it actually works. The first few weeks went the way I suspect they go for a lot of people: one long conversation per feature, a growing pile of context, and a moment near the end where the agent cheerfully announces it’s done and I discover, ten minutes later, that “done” meant “the parts I remembered are done”.

Trying to resume progress on large features led me to try different frameworks such as Spec Kit from GitHub, OpenSpec, the Kiro IDE, standalone Claude skills and so on. I even built my own harness-agnostic spec-driven framework which I used for a while with (slow) success with GitHub Copilot CLI, Codex and Claude Code. That level of ceremony was right for less capable models, but the picture evolved: the models got better at producing what I wanted without being walked through every step, and I found I needed a different kind of help.

What I needed wasn’t more guidance for the agent. It was less trust in it. Generated code reads fluently, compiles, and is sometimes wrong in ways that only show up later: a concurrency guard that guards nothing, a test that faithfully asserts the bug. Fluency and correctness are uncorrelated, and anyone who has reviewed enough agent output has caught that gap personally. Two problems kept coming back:

Long conversations drift. The further a session goes, the more the early decisions fade, and the agent starts re-solving problems it already solved, or quietly undoing them.
“Done” was decided by the agent. There was no external definition of done that the agent couldn’t argue with, so every finished session still needed me to establish, from scratch, whether it was actually finished.

The second one matters more than it looks. When an agent writes the code, the review doesn’t go away; it moves, and it gets harder, because I’m now checking work I didn’t build the understanding for. Review is the scarce resource. Anything that makes the agent produce more code without making that code cheaper to verify just moves the bottleneck around.

What I ended up with is a small (well… not so small anymore) tool called asl, short for agent session loop. The idea is simple: write a plan, cut it into sessions that each fit comfortably in one fresh conversation, attach to every session a check that a program — not the agent — runs, and then let a runner go through the sessions one at a time while I do something else. This post walks through that pipeline, why each step exists, and what I learned building it.

Why plans, not prompts

Most of the leverage happens before any code is written. A vague prompt to an agent yields a confident implementation of something adjacent to what you wanted; a precise plan yields something much closer. So the pipeline starts with a plain Markdown plan, and with a step whose entire job is to make that plan better before anything runs.

The bar I hold a plan to is that it should be decision-complete: every choice the implementer shouldn’t be making on their own has already been made. The test I use is a slightly hostile question: could two competent implementers follow this plan and end up with meaningfully different behaviour? If the answer is yes, the plan is hiding a decision — about an API, a data format, an error case, a migration order — and the agent will happily make that decision for me, silently, somewhere in the middle of a diff.

It helps to separate facts from decisions. “Which callers use this function?” is something the agent can discover by reading the repository, and it should. “Which of those callers are we willing to break?” is a judgement call, and it stays with me. A good plan records the first kind as discovery and the second kind as an explicit decision.

The step that enforces this is a Claude Code skill called plan-iterate. It reads the plan, assesses how complete it is (scope, interfaces, tests, risks), asks me targeted questions about the weakest areas, and edits my answers back into the file. With --ideal, it first does one research pass over the codebase and describes what the unconstrained ideal version of the change would look like, then ranks the gaps between my plan and that ideal by impact, and asks me to adopt, defer or reject each one. Deferred items get written into the plan so the decision is recorded rather than forgotten.

This is the same instinct behind systematic reviews: it’s cheaper to catch a bad idea on paper than in a diff.

The pipeline, step by step

Here is the whole flow, from a plan to a finished run:

Mermaid source of this diagram

The steps in the middle used to be separate commands I had to remember to run in order. These days a single asl prepare PLAN.md --cwd REPO_ROOT runs split, validation, expansion, baseline and the readiness gate deterministically and, on success, prints the exact asl run command to use next.

Splitting: where to cut

Splitting turns one plan into a directory: a manifest.md listing the sessions and their readiness state, plus one session-<id>.md per session. Each session has a scope, the files it’s expected to touch, a handoff section the agent fills in when it’s done, and a completion contract (more on that below).

Where to cut took me a while to get right. A good session has:

One primary outcome, with evidence that can fail on its own. If I can’t write a check that distinguishes “this session worked” from “this session didn’t”, it isn’t a session yet.
A useful state if the later sessions never run. If the run stops after session 3, sessions 1 to 3 should still leave the codebase in a coherent, reviewable state, not half of a refactor.
Room to work in one fresh context. Context is a split criterion in its own right: the files involved, their size, the instructions that apply and the handoffs from earlier sessions all need to fit, with space left over for the agent to think. It’s much cheaper to re-split before running than to discover forty minutes in that a session can’t hold everything it needs.

Two anti-patterns I now avoid on purpose: splitting implementation from its tests (the test session ends up testing whatever the implementation happened to do), and a “cleanup later” session whose absence would leave the change unsafe. Sessions are ordered by dependency, not by whatever is easiest to edit first: shared contracts and scaffolding before their consumers, additions before removals.

Expanding: the plan is a set of claims

Every path, function signature, command and test name in a plan is a claim about the repository, and plans are often written against a mental model that’s a few commits out of date. Expanding is where the agent reads the actual codebase and checks each of those claims: do these files exist, do these functions have the signatures the plan assumes, is this session secretly three sessions?

The rule that makes this useful is that corrections go into the plan, not into the implementation. Problems get written into a ## Contract Issues section of the session rather than silently patched around. If the agent compensates for a stale fact while implementing, the only record of that decision is the diff, and the next session inherits the stale fact. A session that turns out to be too big can be re-split into child sessions with session-resplit while the rest of the plan is left untouched.

Preflight: the gate

Preflight produces red, yellow and green findings, each with a machine-readable repair hint that names the failing file, the check that failed and the smallest repair. A red or yellow finding blocks the plan from becoming loop-ready, and asl run refuses to start on anything else. When there are findings, a companion skill, plan-preflight-iterate, walks through them with me as a short Q&A and reruns preflight until it’s clean. Because each finding names its smallest repair, that pass only touches what broke.

Preflight also checks the contracts themselves: every expected file either exists or is explicitly marked as new, every command is available, and no placeholder or open decision survived into a session.

One design rule from this part spread to the whole tool: keep authored state separate from derived state. The plan, the sessions, the decisions and the handoffs are authored; the manifest’s readiness flags and status tables are derived, and get rebuilt from the authored files. When the two disagree, the tool pauses rather than picking a winner. A readiness table can always be regenerated; a decision that got silently overwritten can’t.

The point of all this ceremony is to move failures earlier. A missing file discovered during preflight costs a few seconds; the same missing file discovered forty minutes into a run costs a confused agent, a half-finished session and a messy diff.

Completion contracts: who decides what “done” means

This is the part I’d keep if I had to throw everything else away.

Every session file ends with a block like this:

 markdown
 
 
 
## Completion Contract

### Done when

- The `export` command writes one JSON file per project.
- Existing CLI tests still pass.

### Verify

```json
{
 "session": "02",
 "checks": [
 {
 "name": "unit-suite",
 "type": "command",
 "argv": ["python3", "-m", "unittest", "-q"],
 "timeout_seconds": 180
 }
 ]
}
```

“Done when” is for humans (and for the agent to aim at). “Verify” is for the runner: a list of checks — commands to run, files that must exist, patterns that must appear in a file — that the Python runner executes after the agent says it’s finished. If a check fails, the session isn’t complete, no matter how convincing the agent’s summary was. Commands are given as an argument list and run without a shell, so there’s no quoting to get subtly wrong.

Writing these contracts taught me that a check is only as good as what it can falsify. A test suite that would also pass for a plausible wrong implementation isn’t evidence of anything, whoever wrote the code. Agents make that old truth urgent, because they produce plausible wrong implementations at volume. So before I accept a contract, I try to describe one implementation that would be wrong but would still pass it. If that’s easy, the contract needs a sharper check, or the session is claiming more than it can prove.

Three details make this more robust than it sounds:

Baselines. Before the run, asl baseline executes every contract and records what it reports on the unchanged codebase, and preflight won’t let a plan start without that evidence. A check can declare how it should behave before any work happens: a reproducer is expected to fail (proving it actually reaches the thing being changed), while protected behaviour is expected to pass throughout. A protected check that already fails is a broken contract, not a goal — and a reproducer that already passes means the session is either done or testing the wrong thing. Either way, better to know up front than to blame the session for a pre-existing red.
Every check backs a named claim, with a limit. “The focused test passes” says nothing about the database, the deployed service or the code paths it doesn’t exercise. Keeping that limit written next to the check stops a green session from quietly turning into “it works”.
The split between judgement and enforcement. The agent decides how to implement a session. Python decides whether it’s done. The skill the agent runs for each session, session-run, is under a hundred lines of instructions, and that’s on purpose: when the runner launches it, Python already holds the plan lock and has checked readiness, and after the agent returns, Python runs the verification, syncs the progress journal and refreshes readiness. Nothing that matters is left for the model to remember.

That last point turned into a design rule for everything else: anything that must happen is code; only things that require judgement go in a prompt.

What a run looks like

Starting a run looks like this:

bash

asl run ~/.local/share/agent-plans/export-command/ --harness claude

The runner then loops:

Re-read the plan files from disk.
Pick the first pending session.
Start one agent turn, in a fresh context, running /session-run on the plan.
When the turn ends, run the session’s Verify checks.
If they pass, mark the session complete and go back to step 1. If not, put the session back to pending and hand the failing evidence to a repair turn; when the repair budget runs out, stop with a reason.

Re-reading from disk on every turn matters more than it looks. The plan directory is the single source of truth, so I can edit a future session while a run is in progress, and the next turn picks up the change. Nothing about the plan lives only in the runner’s memory.

Fresh context, durable handoffs

Every session starts in a brand new conversation, loaded with only what it needs: the instructions that apply, the plan, its own session file and contract, and the handoffs from the sessions it depends on. That’s the fix for drift: there’s no long transcript for early decisions to fade out of.

It also means the handoff has to carry everything. When a session finishes, the agent fills in its handoff: the files it modified, the key decisions it made, anything blocking the next session, and notes such as why it reran a check. Two optional sections are worth singling out:

Discovered facts record what implementation proved about the plan’s assumptions (“the config loader already caches; no need for a second layer”), so the next session doesn’t spend its context re-researching a settled question.
Open questions hand forward what’s still uncertain, named, instead of leaving it to be rediscovered.

My test for a handoff is blunt: if losing the transcript changes what the next session is allowed to do, the handoff is incomplete. Conversations are disposable; the plan directory is the memory.

Pausing is a feature

Agents fail in loops more than in crashes: rereading the same files, rerunning the same green check, restating the same hypothesis as a “new” fix. Left alone, an unattended run will happily burn an hour that way. So a session that can’t make progress doesn’t keep trying; it pauses, and a pause is a durable state transition with a stable reason, not an abandoned conversation.

The vocabulary is deliberately small and fixed: blocked for a decision only I can make, unsafe when continuing could do damage, verify-failed when the contract didn’t pass after repairs, no-contract or malformed-contract when there’s nothing trustworthy to verify against, a guard violation, and so on. The agent isn’t allowed to invent new reasons. Each pause records what blocked it and the evidence, so the next step is obvious. That makes “why did it stop?” a question with an answer, and it means a retry has to come with something new — a changed hypothesis, a fixed fixture, a decision — rather than the same attempt with more hope.

It took me a while to stop treating pauses as failures. A necessary pause is the system working. A run that pushes through an unresolved decision to keep its completion rate up is worse than one that stops and asks.

The controls I use all the time

--harness picks the agent: Claude Code, Codex or Copilot CLI. The same plan runs on any of them, and I can switch mid-run for the remaining sessions with asl control select-harness. A few other controls I use all the time:

asl control stop-after-current lets the current session finish and then exits cleanly, so the next asl run picks up where it stopped. It’s the polite version of Ctrl-C.
--max-sessions N runs at most N sessions and stops successfully. Useful when I want to look at the first session’s output before trusting the rest.
Pauses with questions. If an agent hits a decision it shouldn’t make alone, the session pauses with reason: blocked. asl questions PLAN_DIR renders the questions, and asl run PLAN_DIR --reply "..." resumes the paused turn with my answer.
--review-harness codex adds a final session where a different agent reviews the finished work and writes its findings to a final-review.md.

A second agent, reviewing cold

That last control deserves a little more space. The reviewer runs on a different harness from the implementer, isn’t allowed to change any code, and reviews cold: from the plan, the contracts and the actual changes, not from the implementer’s account of them. Its job is to find where the change violates the intent, the protected behaviour, the scope or the limits of its own evidence. Its output is structured, not “looks good”: every finding has a severity, the files it points at, an explanation, what “fixed” means, and the checks that would prove it, and the run refuses a review whose summary line and findings don’t agree.

Repairs for its findings are bounded too. Findings are grouped, each group gets a capped number of repair attempts, and the review-and-repair cycles themselves are capped, so a reviewer and an implementer can’t ping-pong forever. The principle I hold myself to on top of that: a repair fixes the finding inside the reviewed boundary, and anything that would change the design goes back to the plan as a new session instead of being absorbed into the fix. The cap limits rework; it never lowers the bar for acceptance.

Having one model check another’s work catches a surprising number of things, but I try not to over-read it. A different model brings different habits for finding failures, not independent evidence. What actually decides acceptance is still the checks, the reproducers and me reading the diff.

When the run ends, it ends with an explicit reason: all-complete, max-sessions or stopped for the happy paths, or a named pause like a failing check or a guard violation. There is no “it just stopped”. And “all-complete” still means the contracts passed, not the change is correct: it’s the point where I go and read the diff, the handoffs and the review, not the point where I merge.

Guardrails, briefly

Once a run is unattended, you need a way to steer it, and the most important distinction I ended up drawing is between advice and enforcement. Telling an agent not to deploy is steering; making sure it can’t reach the deploy credentials is a boundary. A prompt expresses intent, never authority.

Guidance is advice. I can send a running session a note (“prefer the existing date helper over adding a dependency”) and it’s delivered into the conversation. It can shape implementation choices, but it can’t change the session’s scope or its completion contract.
Only a guard enforces. asl control guard add can deny a command (matched as an exact argument prefix and enforced by a per-run PATH shim) or protect a path (compared against a snapshot taken at the start of the turn). A violation rejects the turn and pauses the run. It never rolls anything back, so I can always see exactly what the agent did.

A guard also reports, per harness, whether it is active, degraded or unsupported. Not every agent CLI exposes the same hooks, and I’d rather have a guard say “I can’t enforce this here” than quietly imply it did.

There’s more to supervising a run than this, like a mailbox that lets a running agent ask questions without stopping, and a separate “overseer” that watches runs across projects. They all follow the same rule: an AI that can be argued into an action must never be the guard.

When this is overkill

Most changes don’t need any of this. A local, reversible edit with a test that already covers it needs a boundary, a check and a review — that’s it. I reach for a full plan and an unattended run when a change spans several sessions’ worth of work, when the order of the steps matters, or when I want the run to make progress while I’m doing something else. The pipeline exists so every decision has a home when a change is big enough to need one, not so every typo fix carries maximum ceremony.

A quick way to tell: before handing a task to an agent, can I state the observable outcome and one behaviour that must not change, bound the files involved, name a check that would fail for a plausible wrong implementation, and point to someone (usually me) who can actually absorb the diff? If one of those answers is “no” or “not sure”, the right next step is usually a smaller precursor — reproduce the bug, inventory the callers, write the failing test — rather than the whole change.

Lessons from four months of building it

asl started as a small loop in a script in early June 2026. Four months later it’s about 180,000 lines of standard-library-only Python with about 270,000 lines of tests, built largely by running plans through the tool itself. A few things I’d tell myself at the start:

More tests than code is not a vanity metric. When most of the code is written by agents, the tests are the specification, and they’re what makes the next change safe. There is also a reliability-stress suite that reruns timing-sensitive scenarios under load and across time zones, because “it passed on my machine at 3 p.m.” is not good enough for something that runs unattended overnight.
Make failures early and loud. Every hour spent on preflight, baselines and repair hints has paid itself back many times over in runs that didn’t die halfway.
Promote automation by maturity, not enthusiasm. Almost every piece of asl went through the same ladder: do it by hand while watching, write it down as a checklist or skill, script the deterministic parts, and only then enforce it in the runner. Skipping a rung always produced something that failed in a way I didn’t yet understand.
Turn each run failure into a control. The best features came from runs that went wrong. Agents pausing the whole run for every small question became a live question-and-reply channel; answers that were accepted but never actually read by the agent became receipts proving the agent consumed them. Each one started as a reproducer from a real run and landed as its own reviewed change — never as a quick patch in the middle of the run that exposed it.
Write down the reason for every stop. Every pause and every exit has a stable, named reason. That made the tool scriptable, and made it possible to build a supervisor on top of it at all.
Don’t trust a harness to enforce what it can’t. Supporting three agent CLIs taught me that they differ a lot in what they can actually guarantee, and that the honest answer is to report capabilities instead of assuming them.
Be suspicious of feeling faster. A 2025 randomized trial by METR found experienced open-source maintainers were about 19% slower with early-2025 AI tooling on their own repositories, while believing they’d been faster. That’s not a verdict on today’s tools, but it’s a good reason to judge a workflow by what passes review rather than by how productive a session felt.
Standard library only has been worth it. No dependency upgrades, no virtual environment to break, and the whole thing installs by symlinking a directory with GNU Stow, alongside the rest of my dotfiles.

Conclusion

The biggest change wasn’t any single feature; it was moving “done” out of the agent’s hands. Once every session has a contract that a program checks, the rest follows naturally: sessions can be small and start fresh, runs can be unattended, pauses become information instead of mysteries, a second agent can review the first one’s work, and I can spend my attention on the plan and the review instead of on babysitting a conversation.

If you work with coding agents a lot, my advice is to start there, and to start small. Pick one bounded task you’d normally do yourself in under an hour. Before handing it over, write down what “done” means and one check that would fail if the agent got it plausibly wrong. Then review the diff against that, not against the agent’s summary. If it doesn’t hold up, you’ve lost one session and gained a concrete reason; if it does, you have your first contract.

Resources and references

External links

From this website

Building the next Facebook in a couple of weeks (as a humble experiment)

Sébastien Lavoie — Fri, 14 Oct 2022 18:49:00 -0500

Introduction

The following is a report that was part of a project I delivered during my BSc in computer science, reformatted from LaTeX to Markdown. While not being utterly interesting (unfortunately, the source code cannot be freely published due to the nature of the work…), it gives a glimpse of my thought process while building this Django website. I have learnt a few lessons from it and maybe, with some imagination, you might too :). To read a nicely formatted version of this article, you can refer to the original report instead.

Meeting the requirements

In order to meet all the requirements when building this application, a few different techniques, strategies and concepts were put to good use. Firstly, all Django views are class-based and rely on Django REST Framework to reduce the boilerplate needed to write the views. One such (simple) view is shown below:

 python
 
 
 
class Profiles(APIView):
 """Used when listing all profiles at once."""

 # allow everyone to see all basic profiles
 permission_classes = [IsOwnerOrReadOnly]

 def get(self, request):
 profiles = UserProfile.objects.all().order_by("-id")
 serializer = ProfileSerializer(profiles, many=True)
 return Response(serializer.data)

All views in turn rely on serializers to parse incoming data and to transform it into appropriate structures that can be used internally by Django. Some serializers are obviously more complex than others, but for demonstration purposes and for the sake of brevity, an example of a class-based serializer is reproduced here:

 python
 
 
 
class UserPublicSerializer(ModelSerializer):
 """Serializer the User model to get back user information that can
 safely be shared around in the application to other users."""

 # We need easy access to the profile image when listing all profiles
 # at once
 profile_image = SerializerMethodField()

 class Meta:
 model = User
 fields = ["id", "name", "profile_image"]

 def get_profile_image(self, obj):
 return obj.userprofile.image.url

The actual implementation details for signals, forms, URLs, other views and components can be explored in the source code submission provided with this report.

When it comes to version control, Git was used religiously with GitHub to host a private repository. The Conventional Commits specification was followed to help create meaningful commit messages.

Defensive coding strategies were used, both in Python and in JavaScript. In the backend, exceptions were raised directly where no appropriate actions could be taken:

 python
 
 
 
def clean_email(self):
 """Verify that the email is available."""
 email = self.cleaned_data.get("email")
 query_set = User.objects.filter(email=email)
 if query_set.exists():
 raise forms.ValidationError("email is taken")
 return email

In the views, conditional statements were sprinkled everywhere where necessary to avoid expansive computations (e.g., processing a profile image when the request is invalid) and to return early from functions:

 python
 
 
 
# ...
if profile is None:
 return Response(
 "Profile not found!", status=status.HTTP_404_NOT_FOUND
 )

if profile.user.id != request.user.id:
 return Response(
 "You do not have the required permissions "
 "to update this profile!",
 status=status.HTTP_401_UNAUTHORIZED,
 )

serializer = ProfileUpdateSerializer(profile, many=False, data=data)
if serializer.is_valid():
 serializer.save()
 return Response(serializer.data)
# ...

On the frontend, a kindred approach was used to assemble robust requests:

 javascript
 
 
 
try {
 const response = await axios.post(
 `${process.env.REACT_APP_API_URL}/auth/jwt/verify/`,
 body,
 config,
 )
 if (response.data.code !== "token_not_valid") {
 dispatch({
 type: AUTHENTICATED_SUCCESS,
 })
 } else {
 dispatch({
 type: AUTHENTICATED_FAIL,
 })
 }
} catch (err) {
 dispatch({
 type: AUTHENTICATED_FAIL,
 })
}

With security in mind, views were by default protected and accessible only to authenticated users unless a statement such as the following allows all users to read an endpoint:

 python
 
permission_classes = [IsOwnerOrReadOnly]

Requirements were also exceeded by using advanced techniques and tools: Docker, Docker Compose, Poetry, React, Redux, Material UI as well as setting up an SMTP backend to send emails to users were some of the additions made to the project to prove this claim – these will be covered in more details in later sections of this report.

While being a simple application considering time constraints and budget, the solution brings some novelty to the table, making it a valid application that is production-ready while also serving as an excellent prototype for a much bigger project. Some interesting features not commonly found elsewhere include:

Being able to chat in real-time with all users of the platform in a time-sensitive context in a way that supports high concurrency with Django Channels and Redis. In other words, messages disappear automatically after a specified delay, making for a special chat room where one lives in the moment.
The application allows users to easily follow others to then be able to quickly search through their posts by filtering them. The added tweak resides in the fact that all posts are given “total points” without “up” or “down” counts being disclosed directly.
Posts can be hidden at any time by users yet remain viewable by showing the author as being anonymous.
The colourful profile list view allows filtering of all the profiles at once by name and shows distinctive information about users, such as the list of people they follow or the list of people who are following them as a non-obstrusive modal (Material UI pop-up).

As will be discussed in the section State of the art in web development, this solution follows best practices and aims to represent the state of the art, both in terms of technical execution and in the contemporary look of the final product, which is brought into the spotlight in the following figures.

Running the application

Installing and running the web application

Accessing user accounts

All accounts are registered with the same password: pass1234.

To log in, simply use one of the following email addresses:

 
 
 
admin admin@gmail.com (superuser for Django admin)
Andrea andrea@gmail.com
Bob uol.grader@gmail.com
Hider hider@gmail.com (hidden profile)
Mia mamamia@gmail.com
Suliman suliman@gmail.com
Tuan tuan@gmail.com

If you set the SEND_ACTIVATION_EMAIL setting to True in the configuration file socialnetwork/settings.py, emails will be sent from the account uol.grader@gmail.com, which is a real account created to manage emails for this application.

Running locally

Backend

To avoid a layer of abstraction as setting up Redis on some systems may be more difficult, Django’s backend is used for the channels by default. Should you require to use Redis instead, please follow the instructions in settings.py:

 python
 
 
 
CHANNEL_LAYERS = {"default": {
 "BACKEND": "channels.layers.InMemoryChannelLayer"}
}
# To use Redis locally, please comment the above line and uncomment the
# following two dicts:
# CHANNEL_LAYERS = {
# "default": {
# "BACKEND": "channels_redis.core.RedisChannelLayer",
# "CONFIG": {"hosts": [("redis", 6379)]},
# }
# }

# CACHES = {
# "default": {
# "BACKEND": "django_redis.cache.RedisCache",
# "LOCATION": [("redis", 6379)],
# "OPTIONS": {
# "CLIENT_CLASS": "django_redis.client.DefaultClient"
# },
# }
# }

Then, a simple way to get Redis running would be to launch it with Docker:

bash

docker run -d -p 6379:6379 redis

Moving on to installing Django and the rest of the dependencies for the backend:

 bash
 
cd backend

# Activate a virtual environment, e.g.:
python3 -m venv .venv && source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Run the server on port 8000 with existing data
python manage.py runserver

Dependencies can also be installed with Poetry thanks to the provided files pyproject.toml and poetry.lock with the following command:

bash

poetry install

To keep the Docker image simple, pip is used to install the requirements, although beware that Poetry’s dependency resolver will be more predictable in a production application.

Frontend

 bash
 
cd frontend

# Install dependencies
npm install

# Run the server on port 3000
npm run start

Access the application at http://localhost:3000.

Using Docker

A docker-compose.yaml has been prepared to launch all the required parts at once (Redis, Django and React).

bash

docker-compose build
docker-compose up -d

Some other useful commands:

 bash
 
docker-compose down
docker-compose logs
docker-compose ps -a

This container could be set up to run in a cloud infrastructure such as Google Cloud Platform inside a virtual machine. This setup goes above and beyond what was taught in this module, though, therefore resources were not allocated to the actual deployment.

Local deployment

An alternative to the (costly and unfunded) deployment on GCP can be done on a local machine, using Docker to containerize the application and a service like Cloudflare Tunnel to allow traffic from the external world to reach the application, effectively converting the machine into a server.

Summary of the steps needed to deploy locally

Buy/use a domain name and configure its DNS with Cloudflare.
Update Django’s settings.py to allow the domain name to reach the Django application – avoiding CORS issue when localhost:3000 tries to connect to localhost:8000 using that domain – (i.e. ALLOWED_HOSTS).
Install the CLI tool cloudflared, log in to your Cloudflare account with cloudflared tunnel login.
Create a new tunnel: cloudflared tunnel create test-tunnel.
Route that tunnel: cloudflared tunnel route dns <T> <N>, where <N> is the web address to use as a sub-domain and <T> is the tunnel ID.
Add this sub-domain to the DNS management console in the Cloudflare dashboard.
Configure Cloudflare to use the web application:

 
# ~/.cloudflared/config.yml
url: http://localhost:3000
tunnel: SOME_ID_GIVEN_BY_CLOUDFLARE
credentials-file: ~/.cloudflared/SOME_ID_GIVEN_BY_CLOUDFLARE.json

Finally, run the tunnel with cloudflared tunnel run and go to the configured website at its subdomain (e.g., name.domain.com).

Functionality implemented

This application implements all the necessary functionality at the account management level with Djoser. In effect, users can:

Sign up for a new account;
Activate their account via email upon creation (this behaviour is disabled by default for convenience but can be reactivated as mentioned earlier);
Reset their password via email;
Log into their account;
Change their displayed name;
Make their profile hidden to other users;
Log out from their account;
Delete their account permanently by validating their password.

User profiles provide the following features:

Number of followers as well as a complete list of followers shown;
Number of people that a user is following as well as a complete list of these people;
Number of points a user has “given” to posts they like;
Number of points a user has “taken” from posts they dislike;
Where the user is located;
Each user can display a biography;
The date at which a user joined the social network;
The latest date at which the user updated their profile;
A “status” field to prominently feature a statement, which is also shown in the profile listing;
A list of posts written by the user, where the owner can create new posts, delete, edit or hide existing posts so they are no longer displayed publicly in their profile nor in the list of public posts;
An image gallery the user has uploaded, where the owner can add or remove pictures;
The ability for each user to delete their account, effectively cascading a delete of their interactions in the application such as their posts, profile image, etc.;
Users can edit the information they share with others;
To avoid profile pictures slowing down navigation – most notably in the profile list view – all images are automatically resized to maximum 300 by 300 pixels when a user uploads a new image;
Finally, users can also hide their profile entirely – doing so will then mark all their posts as being anonymous and they will not be reachable by any other user of the network.

Following the “profile” view, the “profile list” view (linked with the text “Meet” in the navigation bar) shows all public profiles with most of the information users share. To view the details of a profile (such as a birth date and the date of the last edit of a user), it is necessary to be authenticated, otherwise unauthenticated users will be redirected back to the “profile list” view and an error alert will be flashed. Additionally, the number of followers and people users are following are displayed right from the list view and allow to click to retrieve a complete list of people who are followed. The different icons shown on this page really pop out: the more followers a user has, the bigger the number of people shown in the actual icon representation; also, an icon indicating a value of “0” is slightly greyed out for readability and to let users know they won’t be able to interact with these icons.

As a distinctive mark of interest, everything belonging to a user is marked with a fingerprint icon: when a user views their own profile or posts, they will be able to quickly identify that this information is theirs as long as they are logged in.

Posts implement a basic editing view and allow their author to hide them. Furthermore, posts receive and lose points based on the opinions of users in the network and will end up with positive or negative scores, similar to what Reddit and Stack Overflow have done but without splitting the count in either direction.

The “Read” page – to be found in the navigation bar – shows posts from people users follow at the top, allowing to filter the content of posts based on their interests. The total number of points is shown next to each post entry. As a section section on that page, users have the ability to see a list of public posts as well as search for keywords of interest, just like in the first section just described.

On any page that gets longer than 500 pixels when scrolling down, there will be a little arrow icon allowing users to swiftly glide to the top of the page.

When clicking on the avatar icon displayed in the top-right corner of the application, a “drawer” (in Material UI’s terms) will open to allow users to quickly jump to their profile and to log out of the application.

Behind the scenes, administrators and superusers can be created with full privileges to modify and delete other accounts. These special accounts will be hidden by default from the public-facing frontend, although their profile can later be made public and used as any other account on the website.

Last but not least, there is a public chat room where all users of the application can go to have a spontaneous discussion. In fact, the web application is programmed to automatically discard messages that are older than 20 seconds (by default), allowing users to share special moments that are very much unique and time-limited while avoiding being overwhelmed by content. In the event that a constant flow of messages might be entering the chat room at a given time, users will still have the ability to scroll up in the view (assuming they read fast enough). If they do not scroll up, the view will always focus on the latest incoming message so their attention span will be maintained on this very moment. Every time a user reloads the page or goes somewhere else in the application, the content of the chat window will disappear on purpose, because good discussions are meant to happen in real-time on this website. Users will not be distracted by anything else other than a little avatar icon and people’s names so they can know at a glance who is interacting with them… but they will need to have a good memory to remember everything. After all, the idea behind this chat application really is to exchange some good thoughts in the moment. What lives on is only the memory of time well spent with others.

At the API level, users can interact with the application directly via API endpoints, provided they send their authentication token should a view have a requirement in this regard. These endpoints are listed succinctly below, but a file openapi-schema.yml has been provided in the backend directory for a complete reference.

 
 
 
accounts/ profiles/
accounts/ profiles/<int:user_id>
accounts/ profiles/<int:user_id>/image
accounts/ profiles/<int:pk>/followers
accounts/ profiles/<int:pk>/followers/add
accounts/ profiles/<int:pk>/followers/remove
accounts/ profiles/gallery/add
accounts/ profiles/gallery/remove/<int:gallery_pk>
accounts/ ^media/(?P<path>.*)$
api/ posts/<int:pk>/
api/ posts/author/<int:pk>/
api/ posts/<int:pk>/up/
api/ posts/<int:pk>/down/
api/ posts/<int:pk>/pointsby/
api/ posts/

Executing the unit tests

To run the unit tests, it is a matter of activating a virtual environment (as described in the section Running locally) and using the built-in test command provided by Django in the following manner, which must be run from within the backend directory:

 bash
 
 
 
$ python manage.py test
Found 119 test(s).
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
..............................................................
--------------------------------------------------------------
Ran 119 tests in 0.584s

OK
Destroying test database for alias 'default'...

Logical approach

Frontend framework: React

The application was designed with the goal in mind of having a modern frontend framework relying on an excellent CSS framework to provide icons and other components. Speaking of components, React was chosen for its strong focus on designing small components that can be reused. Taking advantage of this approach, some components such as the “profile intro” (the box showing the main information regarding a user profile) are literally displayed on different pages, such as when viewing a single profile or when viewing a list of profiles.

Similarly, a “post” entry represents the exact same React component when viewed in isolation or when browsing for many more posts. When search bars are used, these share the same characteristics and behaviors across the page while managing a different state.

React components and icons: Material UI

On the design side, the presentation of the website was made much more consistent with the help of Material UI, which provides many ready-made React components. The whole application was built around the concept of adding building blocks from Material UI such as a “box”, “button” or “typography” to guarantee that all aspects of the application look homogeneous and with the same modern style.

Components such as a “drawer” and a “navbar” really sped up development and were expected to be used extensively right from the start.

API: Django Rest Framework

Since the use of Django was mandatory for the backend part of this work, Django REST framework was an ideal choice considering the need for an API to be directly accessible to end-users and to benefit from an easy-to-use toolkit to build an API that can be accessed from React without breaking a sweat.

Django REST Framework simplifies data serialization and even allows dealing with nested serialization in a straightforward way. Because it comes with powerful features for authentication and permission management, it was also a great option for this specific use case which depend easily on these sets of features which would arguably be very complex, error-prone and time consuming to implement manually.

User management: Djoser

Beyond Django’s default user management system, there was a desire to extend the default behaviour, for instance to use an email address instead of a username to authenticate users. Djoser makes these changes and much more a lot more simple, adding easy management of logins, logouts, password confirmation resets, activation emails and so forth.

By leveraging Django’s built-in features with Djoser, the user model can be extended with ease and flexibility.

Asynchronous communication

To build the chat application, Channels and WebSocket were considered as one of the best combination of libraries because they are simple to use and allow high concurrency to be possible. Channels makes it easy to create an asynchronous WebSocket consumer to deal with many users connecting to possibly many different groups: it is therefore an ideal option for a growing social network.

The WebSocket library in JavaScript is then used to communicate with Channels via the WebSockets technology, sending and receiving messages with additional payload to extend the functionality of the chat application.

Deployment: Docker, Cloudflare Tunnel

To deploy an application to production, Docker Compose is a robust and reliable solution, especially for this use case where different microservices can be pieced together seamlessly, sharing resources across a single network on different ports.

Docker makes it possible to containerize parts of the application (i.e., the backend, frontend and in-memory cache database) to be run potentially on many virtual machines in the cloud and Docker Compose makes the process of orchestring an application such as this one much more accessible than configure web servers from start to finish.

Poetry, a package and dependencies management system for Python, allows requirements to be pinned to specific versions and ensures reproducibility is achieved, which is critically important when it comes to deploying to production. Its dependencies resolver is much more reliable when the exactitude of a list of requirements must be met.

In terms of deploying a real application on a small scale, Cloudflare Tunnel was also considered as it is useful to share prototypes during development and to instantly deploy a local server. Furthermore, it can work hand in hand with Docker Compose and Poetry since a container can be run in the background and the application can be exposed on multiple ports locally without having to configure firewalls.

Because pip is traditionally used in the Python world to manage requirements, the command pip freeze was used to generate a standard list of pinned dependencies, although for an actual deployment to production, an open source alternative like Poetry would likely be preferable.

Formatting: Prettier, Black, Vulture, Pylint

While code formatting is of utmost importance for readability, it is also a tremendous waste of time to try to do it manually. For this reason, multiple tools were used to aid with this gargantuan task. Prettier was used on the frontend side to format everything related to JavaScript and CSS.

On the backend side, Black was the primary formatter used. Its use was complemented with Pylint, which helps to clean up and refactor the code by finding hard-to-spot warnings, such as unnecessary return statements or the use of more complex expressions when an equivalent such as a list comprehension could be used.

Finally, to keep the code tidy and alive, Vulture was used to find “dead” code not in used anywhere else. Even if code is well formatted, code that is not needed become a liability. In the source code world, less really is more.

API calls: Axios

The Axios JavaScript library was used for the attractiveness and flexibility of its API. Because it supports fully the Promise API and allow making asynchronous calls with the async/await syntax, it makes for a good tool to communicate with the backend. It has extensive documentation and excellent support on various web platforms including GitHub, which is an important consideration when thinking about deploying an application in the real world.

There are new and shiny technological options all the time, but a battle-tested software like Axios remain a competent tool apt for the job.

State management: Redux and Immer

Despite the Context API from React being powerful and easy to use, Redux and Immer really make dealing with state management a breeze. Redux on its own is one of the best in its category to provide state management at scale: its documentation is also fantastic and detailed and its popularity on Stack Overflow is only a plus in its favour should there be a need to fix issues while using it (because there will inevitably be such a need). The Redux workflow with actions being dispatched with types makes sense and adding new features is very fast without having to change existing pieces of states when combining reducers.

One possible drawback of using plain Redux to manage the store is that states can be mutated and unexpected consequences may ensue. For this reason, Immer was included so that dealing with a Redux store is stripped down to a direct object assignment, where Immer takes care of updating items and even nested items as needed.

Design and implementation decisions

Because deployment with Docker was a primordial concern, the application was built with a backend that is completely decoupled from a frontend. In fact, with just a little bit of ingenuity, Django could be swapped entirely with another web framework as long as the same endpoints are provided to the frontend with the necessary technologies (such as web sockets) being implemented. In a similar way, the frontend, which is built primarily with React, could be upgraded to another popular alternative such as Vue.js. This gives flexibility and should there be a greater need for optimization once the social network takes off for a real spin, there is the possibility to bundle React with Django by building minimized versions of the assets and shipping everything as a Docker container that can be run on powerful machines in the cloud.

Beyond this original concern, more concrete decisions were taken. For instance, Django signals were used to simplify how user profiles are created. Instead of allowing users to create a profile from their user account, the experience was unified by transparently automating this process whenever a user account is created such that it is attached to a newly created profile associated with it. Leaving the option to the end-user of manually creating a profile could be confusing at best and lead to unexpected results at worst, where user accounts and user profiles IDs become out of sync, which is a nice side-effect of having them created as well as deleted at the same time.

From the start, Djoser was chosen to override specific built-in functions performed by Django and to facilitate email management. To integrate Djoser with the administrative bits of the application – namely, the Django admin interface – Django forms were used to customize how users are to be registered as well as superusers and staff accounts. By overriding the user creation, it is now possible to update parts of the process with simple configuration variables implemented by Djoser, such as forcing users to retype their passwords or not.

With Django models, the base model became a UserAccount, which allow users to join the network. However, the functionality related to account management is kept separate from other features of the application by interfacing with the accounts using a UserProfile model on which a user field exist. In turn, this model integrates other parts of the models, such as many-to-many relationships with followers and a gallery image model, which is also linked as a many-to-many relationship. Another part of the application implements a Post model where the author field corresponds to a UserAccount, effectively linking a different part of the application to the user accounts. Again, a Post then interfaces with other models to take points into account, which are also linked to user profiles.

To manage all kinds of POST requests as well as to guarantee that models communicate as expected between each other, serializers using Django REST framework features were used in the process. These generally use a SerializerMethodField to define more complex behaviors, such as getting all non-hidden followers for a single profile.

Views were implemented with DRF APIView, bridging the gap between a RESTful API and the ability to define simple views using methods like get and delete. Views rely on serializers and specify a permission_classes list of custom values, which were implemented separately in a permissions.py file. This allows customizing the default behavior so that by default all views require authentication unless explicitly stated otherwise by these permission classes. This is also more secure, preventing users from inadvertently being able to access what should have been protected resources.

In terms of API definition, simplicity was key: every endpoint regarding authentication is accessed under /auth while our specific implementation of user accounts is accessed under /accounts. Additionally, all endpoints to deal with user posts are to be found with an endpoint starting with /posts. Finally, there is another set of endpoints to work with profiles, these being accessible under /profiles.

The Django chat application, due to its asynchronous nature, uses different technologies behind the scenes, including notably WebSockets. Channels was chosen to implement this portion because it extends Django’s capabilities beyond HTTP and is ideally suited for chat applications like the one being built as part of this social network.

Evaluation of the final product

What worked well

Opting for React made for a flexible framework that behaves extremely well with Django. React has complete documentation and extensive support offered on different medium, making it ideal for a robust and complex web application. Coupled with Material UI, the frontend aspect of the work was taken care of and didn’t need particular attention when developing. For any basic functionality that was not available out of the box in JavaScript, the Lodash library was used to make object comparisons and such much more readable and understandable when multiple conditions were being evaluated in a JSX expression, often with ternary operators.

The state was managed beautifully with Redux and the Redux Devtools web browser extension, allowing for rapid iterations and feedback where debugging would otherwise become very cumbersome. The useState hook was used profusely to separate concerns as well.

Keeping with React, the application was divided into logical sections that made it easy to add new features. Small components were created for different parts of the application, Redux actions and reducers were stored in their respective directories, containers and other higher-order components were given a special place in the frontend hierarchy, utility functions were well organized and accessible, pages and custom hooks also had their own place to reside.

Deploying with Docker remained a clean process, where a single Dockerfile was created to power the backend and another Dockerfile added for the frontend. Finally, all these instructions were put together with a readable docker-compose.yaml file, which purpose was to join all the different containers in a single place and to allow adding more microservices, such as Redis. Postgres was not used in this version for practical reasons when sharing the source code of the web application with others, but only a few steps would need to be taken to update Django’s settings file as well as pulling from DockerHub a suitable image to run Postgres.

Testing with Django went particularly well: the documentation had a clear guide about testing and any aspect of the application – serializers, views, models, forms, consumers, signals and so on – were tested according to the Django standards, leading to a whopping total of 119 passing unit tests, which are to be found inside a tests directory inside each Django application within the project.

What could have been improved

Planning of the technical stack could have gone much better. During the experimentation phase in the early steps of this project, good portions of functionality were implemented using Django templates without any frontend framework, except for the CSS framework Bootstrap. Features such as user notifications, personal messaging between two users with the ability to use different “inboxes” and “threads” was a very good selling point of the first version of this web application. Alas, due to time constraints, these useful additions were left behind (preserved with a Git history) but did not manage to make it to the final product. It would have been much more productive to take a closer look at the panoply of requirements to be implemented from the very beginning and determining the best fit as early as possible to avoid rewriting code from one framework to another.

Although Redux was a lifesaver in many circumstances, it was also a hurdle in terms of productivity because of its complexity and different moving parts. Being more careful to not use Redux at all costs everywhere possible, dispatching updates in the application would surely be simplified with the Context API using pure React or even some useEffect hooks, which were used profusely with good results. Immer was also introduced a bit later in the thought process when dealing with nested JSON objects. Although it made the remaining development easier, it was also probably a red flag meaning that the Redux store could have been simplified into smaller reducers where different selectors could access parts of the store that wouldn’t need to be nested objects. This may also indicate a flaw in the way models were implemented, but most likely there were errors made when dispatching messages in a part of the Redux store that really should belong somewhere else. In concrete terms, dispatching an event regarding users inside the “user” reducer was probably more complicated than simply dealing with all notifications in a different reducer. In actuality, this led to some dispatch actions that did more than one thing, which is not ideal when thinking about the concept of separation of concerns.

Although this is not perceptible in the final product, Django migrations were at times a pain to work with. This is because the models weren’t thought through very well at the beginning and despite the iterative process being an inherent part of software development, iterating over models in Django definitely counts as an exception to the rule where clearly defined models will not cause migrations to break or need to be modified in any way. It would have been more productive to produce well-designed diagrams, depicting the architecture of the application as well as the complete schema of the database to avoid surprises down the road.

State of the art in web development

On a technical aspect, many parts of this web application conform to modern web standards. The latest stable release versions of libraries were used whenever convenient and/or possible. This non-exhaustive list would include, at the time of this writing, React 17.0.2, Redux 4.1.2, WebSocket 1.0.34 and Material UI 5.5.0 for the frontend. On the backend, Django 4.0.3, Channels 3.0.4, Djoser 2.1.0 and Redis 4.2.0 were used.

When dealing with dependencies, Poetry was used with great benefit to make sure that Docker containers could be deployed reliably. Using Docker and Docker Compose was another touch that was added since most contemporary software will ship with a Docker solution for convenience. An excellent example in this regard would be PlantUML, a Java-based application to generate diagrams from textual descriptions that can be deployed in only a few seconds via Docker, launching a local server with the full capabilities offered by the piece of software in question. Nowadays, with Docker containers being more secure and very fitting for a deployment to the cloud at most big technology companies, it really is a best practice adopted by good software teams to work with containerized versions of their products.

The wider community was consulted in an effort to bring the maximum quality possible to the end result. Concretely, advice was taken from prominent figures in the web development world, including Dennis Ivanov and Laith Harb, both Django and React experts and Corey Schafer, an accomplished software developer from the Python world. Nigel Poulton – author of Docker Deep Dive – served as a good reference for the deployment part. Real Python was a godsend with its in-depth articles, for instance about debugging code with Pdb. Less well-known yet high-quality content producers inspired work with Djoser and the implementation of user posts, including Bryan Dunn and Legion Script. Cloudflare was also one of the best resources used to deploy a secure web server locally and to learn about JSON web tokens and their use specifically with Python. Needless to say, the official documentation from each major framework and tool was referenced all way through, many names which are going to sound familiar by now – Django REST Framework, React, Redux, Immer.js, Django, Djoser, Channels, etc.

Demo of the app

Resources and references

Using Google Sheets as a database to extract data converted to Markdown

Sébastien Lavoie — Sun, 19 Dec 2021 21:00:00 -0600

Introduction

The purpose of this tool is to parse and convert the content of a Google Sheets spreadsheet to Markdown. In this specific case, I just wanted a simple script to reduce the friction in getting a valid output I could use to maintain a “learning log” on my website. So that gives me a quick way to input data in a spreadsheet and then I can take the time whenever needed to update the learning progress as desired by reformatting any of the entries and optionally adding notes to them.

General software requirements

This tool should:

Be able to read a private spreadsheet with a service account configured with the Google Sheets API.
Convert rows from a Google Sheets spreadsheet into Markdown (see Input and Output sections below for details).
Skip rendering rows that are missing a value for the Title or Date columns (it couldn’t be rendered in the correct place properly without this information).
- Other columns are optional and the corresponding level of nesting will be skipped if no value was provided (e.g. a category Articles may have an item with no sub-category and could be missing any of the other values except for Title and Date).
Separate entries by month by parsing the Date column.
- A command-line parameter should exist to specify a year from which the date should be extracted.
- The output should be in reverse chronological order (i.e. for a full year with entries for each month, December will appear at the top and January at the bottom).
Output a hierarchical format like the following: Category > Sub-category > Title > Activity & notes
Group entries by category and sub-category (e.g. if a category is named Articles and there is a sub-category named Python, then Python must be nested under Articles for the corresponding month).
Sort on the Date column before doing any parsing on other columns (rows in the spreadsheet can be in non-chronological order).
- Dates from a single year will be kept (either the year received as a parameter or the most recent year found).
Sort Category, Sub-category and Title alphabetically (ascending order from top to bottom).
Not sort the Activity column to preserve the order in which rows were added to the spreadsheet.
Render links only if the Link column is a valid URL.
Not validate the content of columns except for Date and Link (i.e. render the other columns as is).
Display items with no category above those having a category. The same logic would be true for sub-categories.
Format the output:
- Date should become headers (##).
- Category should be bolded.
- Sub-category should be emphasized.
- Title should become a link if the Link column contains a valid URL.
- Activity should be nested under Title, occupying a new line for each activity.
- Notes should appear (italicized and inside parentheses) next to the activity.
  - If there is a value for Notes but no value for Activity, the output would be next to the Title.

Input

A spreadsheet with the following columns (starting with Date):

[additional info]	Date	Category	Sub-category	Title	Activity	Link	Notes
November	13/11/2021	Articles	Python	Python slots, slots, and object layout		https://www.youtube.com/watch?v=Iwf17zsDAnY	understood that X does Y
November	14/11/2021	Books	Software engineering	The Pragmatic Programmer	ch. 1	https://pragprog.com/titles/tpp20/the-pragmatic-programmer-20th-anniversary-edition/	note 1 here
November	15/11/2021	Books	Software engineering	The Pragmatic Programmer	ch. 2		note 2 here too
November	16/11/2021	Books	Software engineering	Clean Code
December	1/12/2021	Documentation	Python	The Python Tutorial	Sections 1-4	https://docs.python.org/3/tutorial/index.html
December	2/12/2021	Books	Software engineering	Clean Code	ch. 2-10
Render available values	2/12/2021	Books		Clean Coder			book note
Render available values	2/12/2021			Python/C API Reference Manual		https://docs.python.org/3/c-api/index.html
Title missing: skip rendering	3/12/2021	Articles
Title missing: skip rendering	3/12/2021	Articles	Software engineering		ch. 3	https://docs.python.org/3/c-api/index.html	note
Date missing: skip rendering		Articles	Python	Python slots, slots, and object layout	Section 2	https://www.youtube.com/watch?v=Iwf17zsDAnY	note not rendered

Output

A Markdown output to the terminal in the following format:

 markdown
 
 
 
## December

- [Python/C API Reference Manual](https://docs.python.org/3/c-api/index.html)
- **Books**
 - Clean Coder (_book note_)
 - _Software engineering_
 - Clean Code
 - ch. 2-10
- **Documentation**
 - _Python_
 - [The Python Tutorial](https://docs.python.org/3/tutorial/index.html)
 - Sections 1-4

## November

- **Articles**
 - _Python_
 - [Python slots, slots, and object layout](https://www.youtube.com/watch?v=Iwf17zsDAnY) (_understood that X does Y_)
- **Books**
 - _Software engineering_
 - Clean Code
 - [The Pragmatic Programmer](https://pragprog.com/titles/tpp20/the-pragmatic-programmer-20th-anniversary-edition/)
 - ch. 1 (_note 1 here_)
 - ch. 2 (_note 2 here too_)

Input/Output example when using this tool.

Installing this tool

Tested only under Python 3.9.7, requires at least Python 3.7+.

Using `pip`

[recommended] Activate a virtual environment.
To install dependencies, run from this directory: pip install -r requirements.txt
Then execute the script to run the tool: python3 get_learning_logs

Using `pipenv`

Run from this directory: pipenv install.
Activate the newly created virtual environment with pipenv shell.
Then execute the script to run the tool: python3 get_learning_logs

How to use this tool

You will need to create an environment file (default path: ~/.learning-logs or edit the path for LEARNING_LOGS_ENV_PATH in get_learning_logs.py). It should look as follows:

 env
 
SERVICE_ACCOUNT_FILEPATH="/path/to/service_account_learning_logs.json"
SPREADSHEET_ID=SPREADSHEET_ID # Found in the URL of the document
WORKSHEET_ID=0 # First sheet is 0 by default, it comes after the URL parameter `gid`

To set up this project, you will have to:

Enable the Google Sheets API for your project (https://support.google.com/googleapi/answer/6158841?hl=en).
In the project, search for “service account” in the search bar and create new credentials.
Download the credentials and put them at the path SERVICE_ACCOUNT_FILEPATH referenced in the environment file.
Create a spreadsheet with the following columns (not necessarily in that order): Date, Category, Sub-category, Title, Activity, Link, Notes.
- The column Date Should match a date format: select all cells from that column, go to Format > Number > Date to apply the expected format (mm/dd/yyyy). You can then input a date (e.g. 11/1/2021) and you will see a calendar pop up when double-clicking on it. Dragging it down to a new cell will create a new date for the next day (in this example, 11/2/2021).
Get its SPREADSHEET_ID (https://docs.google.com/spreadsheets/d/SPREADSHEET_ID_HERE/edit#gid=0) to put it in the environment file.
Get the WORKSHEET_ID (the integer after gid= in the URL of the document).
From the spreadsheet document (or from the folder view in Google Drive), share with the email associated with the service account (e.g. learning-logs-something@project-name-12345.iam.gserviceaccount.com). Read access (“Viewer” permission) is enough.

By now, the script is ready to be called. I just created an alias to execute it more easily by putting the following in ~/.bash_aliases (you will have to adapt the paths of course):

 bash
 
alias learning-logs='~/.local/share/virtualenvs/learning-logs-to-markdown-XJLvhmzn/bin/python3.9 \
 ~/dev/sglavoie/dev-helpers/learning-logs-to-markdown/get_learning_logs.py'

This will output the converted data from the spreadsheet to Markdown as shown above.

Conclusion

This little tool scratched an itch and will be helpful in making the process of updating the learning progress of this website more straightforward, more convenient and less hands-on! You can find its source code on GitHub.

Convenient and lightning fast backups with rsync

Sébastien Lavoie — Sat, 31 Jul 2021 09:01:00 -0500

Introduction

Coming into its fourth year of existence and usage in production environments, this simple script has proven to be a fantastic ally. Despite the plethora of great options out there that have more bells and whistles, I have yet to find a single complaint about this solution as it hinges on the minimalist side, requiring only a working Python 3 installation along with the powerful rsync command-line tool.

It’s one of those pieces of code that perform some really basic tasks that can be scheduled with cron in the background, that one forgets about until a real need to access a backup comes up. This script is barely a Python wrapper calling rsync but still, I felt like sharing it for those who like uncomplicated setups. I’ve literally left this script alone for years some time after the release of Python 3.6 which was released back in December 2016 and it has been going strong since then mostly due to the fact it does not do much more than backing files up and leveraging rsync.

Prerequisites

As briefly mentioned above, this tool uses Python 3.6+ and rsync. That’s it, really.

Setting up the script

This script is being made available on GitHub within a set of other small tools. To get the source code for this particular tool, you can either clone the whole repository, get the content of the files separately or simply copy and paste the current version from below. You will need a settings.json file as well as the Python script that will run the necessary commands.

The configuration file

Right off the bat, you will most probably want to update the array of strings for the data_sources key as well as the string for the key data_destination in the following JSON file.

 json
 
 
 
{
 "data_sources": [
 "/home/sglavoie"
 ],
 "data_destination": "/media/sglavoie/Elements",
 "terminal_width": 40,
 "sep": "=-",
 "log_name": ".backup_log_",
 "log_format": "%y%m%d_%H_%M_%S",
 "rsync_options": [
 "-vaAHh",
 "--delete",
 "--ignore-errors",
 "--force",
 "--prune-empty-dirs",
 "--delete-excluded"
 ],
 "backup_exclude": ".backup_exclude"
}

By default, the following options are passed to rsync:

Option	Description
`-vaAH`	verbose, archive, ACLs, hard-links (preserve)
`--delete`	“delete extraneous files from destination dirs”
`--ignore-errors`	“delete even if there are I/O errors”
`--force`	“force deletion of directories even if not empty”
`--prune-empty-dirs`	“prune empty directory chains from the file-list”
`--delete-excluded`	“also delete excluded files from destination dirs”

Description of available settings

Name of setting	Description
`data_sources`	Directories to backup, supplied as a list of strings (no slash at the end).
`data_destination`	Single destination of the files to backup, supplied as a string. This can be overridden when passing option `-d` or `--dest` to the script
`terminal_width`	Line length in the terminal, used for printing separators.
`sep`	Separator to use along with `terminal_width`.
`log_name`	Sets the prefix of the log filename.
`log_format`	This goes right after `log_name` as a suffix.
`rsync_options`	Options to use with rsync as a list of strings.
`backup_exclude`	Default file in each source in `data_sources` where files/directories will be ignored.

The script

 python
 
 
 
"""
Script that uses `rsync` to make a simple and convenient backup.
Note: requires Python 3.6+. No other Python third-party libraries required.
"""
import argparse
import datetime
import glob
import json
import os
import pathlib
import subprocess


def run_backup():
 """This is where all the action happens!"""
 settings = get_settings()

 # initiate the parser
 parser = argparse.ArgumentParser("backup")
 parser.add_argument(
 "-c",
 "--clear",
 help="Delete all log files for current source in DATA_SOURCES.",
 action="store_true",
 )
 parser.add_argument(
 "-s",
 "--src",
 dest="source",
 default=None,
 help="Specify an alternative source to backup as a string.",
 action="store",
 )
 parser.add_argument(
 "-d",
 "--dest",
 dest="destination",
 default=None,
 help="Specify an alternative destination for backup as a string.",
 action="store",
 )

 # read arguments from the command line
 arguments = parser.parse_args()

 # check for --source or -s
 # Replace potential list of sources to backup with this one
 if arguments.source is not None:
 if os.path.isdir(arguments.source):
 settings["data_sources"] = [arguments.source]
 else:
 print("Please enter a valid source to backup.")
 exit(0)

 # check for --clear or -c
 if arguments.clear:
 clear_logs(
 data_sources=settings["data_sources"],
 log_name=settings["log_name"],
 )

 # check for --dest or -d
 # Replace destination to backup with this one
 if arguments.destination is not None:
 if os.path.isdir(arguments.destination):
 settings["data_destination"] = arguments.destination
 else:
 print("Please enter a valid destination.")
 exit(0)

 # don't run the script if the destination doesn't exist
 if not os.path.isdir(settings["data_destination"]):
 print(
 f"The destination doesn't exist.\n({settings['data_destination']})"
 )
 exit(0)

 backup_all_sources(settings)


def backup_all_sources(settings: dict) -> None:
 """Iterate over all sources to backup."""
 for source in settings["data_sources"]:
 date_now = datetime.datetime.now()
 log_format = datetime.datetime.strftime(
 date_now, settings["log_format"]
 )
 log_filename = f"{source}/{settings['log_name']}{log_format}"
 log_option = f"--log-file={log_filename}"

 backup_source = settings["backup_cmd"].copy()
 backup_source.extend([log_option])

 # files to ignore in backup
 exclude_file = f"{source}/{settings['backup_exclude']}"

 if os.path.exists(exclude_file):
 exclude_option = f"--exclude-from={exclude_file}"
 backup_source.extend(
 [exclude_option, source, settings["data_destination"]]
 )
 else:
 # skips '--exclude-from' option if no file is found
 backup_source.extend([source, settings["data_destination"]])

 settings["source"] = source
 settings["backup_source"] = backup_source
 settings["log_filename"] = log_filename

 try:
 backing_source(settings)
 except KeyboardInterrupt:
 print("\nKeyboardInterrupt: Exiting operations.")
 exit(0)


def backing_source(settings: dict) -> None:
 """Print information to STDOUT and to `log_filename` and executes the
 rsync command."""
 print(settings["sep"] * settings["terminal_width"])

 cmd_executed = " ".join(settings["backup_source"])
 msg_executed = f"Command executed:\n{cmd_executed}\n"
 print(msg_executed)

 with open(settings["log_filename"], mode="w") as log_file:
 log_file.write(f"{msg_executed}\n")

 try:
 child = subprocess.Popen(settings["backup_source"])
 _ = child.communicate()[0] # call communicate to get the return code
 rc = child.returncode
 except FileNotFoundError:
 print(f"FileNotFoundError: Is the `rsync` tool installed?")
 exit(1)

 print(f"\nBackup completed for: {settings['source']} (return code: {rc})")
 print(settings["sep"] * settings["terminal_width"])


def clear_logs(data_sources: list, log_name: str) -> None:
 """Clears log files for each source specified in SETTINGS."""
 for source in data_sources:
 # Retrieve a list of all matching log files in `source`
 log_files = glob.glob(f"{source}/{log_name}*")
 if log_files == []:
 print(f"\nThere is no log file to delete in {source}.")
 exit(0)
 else:
 print(f"Log files in {source}:")
 for log_file in log_files:
 print(log_file)
 if user_says_yes():
 for log_file in log_files:
 os.remove(log_file)
 print("Log files deleted.")
 print("Exiting script...")
 exit(0)


def user_says_yes(
 msg: str = "\nDo you want to delete log files for this source? (y/n) ",
) -> bool:
 """Asks the user to enter either "y" or "n" to confirm. Returns boolean."""
 choice = None
 while choice is None:
 user_input = input(msg)
 if user_input.lower() == "y":
 choice = True
 elif user_input.lower() == "n":
 choice = False
 else:
 print('Please enter either "y" or "n".')
 return choice


def get_settings() -> dict:
 """
 Get the settings from `settings.json`.

 Returns:
 dict: Containing all settings used by the tool.
 """
 directory = pathlib.Path(__file__).parent.absolute()
 with open(directory / "settings.json") as fp:
 settings = json.load(fp)

 backup_cmd = ["rsync"]
 backup_cmd.extend(settings["rsync_options"])
 settings["backup_cmd"] = backup_cmd

 return settings


if __name__ == "__main__":
 run_backup()

How to use

One-time setup

Set all values in settings.json to suit your needs.
Make sure that the backup destination is available/mounted. A simple warning will be echoed if the destination can’t be found.
Call the Python script. As an example, if it’s located at /home/user/backup/rsync_backup.py, then you could put the following alias in ~/.bash_aliases (or equivalent)*:

 bash
 
alias backup='python3 /home/user/backup/rsync_backup.py'

* Python may be called differently on your system, e.g. simply python instead of python3.

In this instance, the script can now be executed in a terminal with the keyword backup along with optional arguments.

Using as a daily driver

Make sure the destination to back files up is available/mounted and simply call the script (e.g. backup in this example).

The output will look something like this:

 bash
 
 
 
$ backup
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Command executed:
rsync -vaAHh --delete --ignore-errors --force --prune-empty-dirs \
 --delete-excluded --log-file=/home/sglavoie/.backup_log_210731_09_28_09 \
 --exclude-from=/home/sglavoie/.backup_exclude /home/sglavoie /tmp

building file list ... done
deleting sglavoie/.backup_log_210731_08_47_24
sglavoie/
sglavoie/.bash_aliases
...

Avoid deleting files at the destination

The rsync command first builds a list of the files that are part of the backup and will go on adding new files, updating existing files as well as removing files that are found in the destination but not in the source anymore according to the default list of flags passed to it in this implementation. It is therefore important to note that if you would like to back up files in the destination that no longer exist in the source, a convenient solution is simply to create a new folder at the root level inside the destination directory and manage those files separately.

For instance, I sometimes want to back up large video files but there is rarely a need to update them. In this case, it would be possible to add the directories to exclude in the file .backup_exclude (default configuration option) like so:

Videos/
Dropbox/other/folder/

Then, these directories will be ignored when backing up the source (if they are nested inside the source directory to start with!). If the destination is set to /media/user/backup and the source set to /home/user/my_folder, then the script would recreate the “root” directory of the source if it doesn’t exist on the destination:

bash

$ ls /media/user/backup
my_folder

You could keep track of those heavy files that do not need to be backed up frequently by putting them manually next to my_folder in /media/user/backup. This will lead to huge speed ups when running the script if you track many files in this way!

Conclusion

This was a quick overview of what rsync can offer: I neither touched upon its remote syncing capabilities nor the vast majority of its options, for which you can find a lot more information about by typing man rsync to display the manual page in the terminal.

Without making use of any fancy features of recent versions of Python or rsync, I have found this simple backup procedure to have worked flawlessly for a long time. Hardware inevitably fails at some point and after having heard about countless examples of people losing some or all of their most precious files (including my own story…), I have come to enjoy backing my system with this little script.

I wouldn’t trust my external drive to last forever either, hence I also rely on Dropbox and Google Drive for documents I find are a good fit. But overall, I just do not tend to back up every single system file in the cloud, so there is still a convincing use case where daily and complete backups become a possibility when using a fast approach like rsync to incrementally save snapshots.

More resources and references

rsync official website – samba.org.
Keeping Linux files and directories in sync with rsync – RedHat, going through the main options offered by the tool.
How To Use Rsync to Sync Local and Remote Directories – DigitalOcean, demoing how to use the tool and covering the case of remote file access.

Black: A Python Formatter to Brighten Your Workflow

Sébastien Lavoie — Sat, 13 Jul 2019 13:10:00 -0500

What is Black?

From its GitHub repository:

Black is the uncompromising Python code formatter. By using it, you agree to cede control over minutiae of hand-formatting. In return, Black gives you speed, determinism, and freedom from pycodestyle nagging about formatting. You will save time and mental energy for more important matters.

Black will take less than ideal code and will automatically reformat it according to the PEP-8 style guide. It will take care of many things, including the following (taken in that order from README.md in Black’s repo):

Wrapping lines properly by shortening as much as possible single statements;
Break long lines;
Remove superfluous empty lines;
Take care of adding trailing commas where necessary;
Convert single quotes (') to double quotes (");
Convert numeric literals to lowercase (0XAB to 0xAB);
Superfluous parentheses are removed;
Split call chains on different lines;
Works from the terminal and support many code editor integrations (Emacs, Vim, VS Code, Sublime, etc.).

Now that it’s clear why it can be a good idea to use it, let’s see how it works with Vim and the terminal.

How to use it?

First, we need to install it using Python 3.6.0+:

bash

$ pip install black

That’s it!

Use it from the terminal

In its simplest invocation, we can use Black by specifying a file or a directory by typing:

bash

$ black path_to_file_or_directory

It doesn’t come with many options as its goal is to automatically format code consistently, but you can find out more about this tool in the following way:

bash

$ black --help

Use it with Vim

Using Black as part of your workflow in a code editor is where I believe it really shines, because you can see almost in real-time when you make a formatting mistake, which will help down the road in committing less sins.

Integrating Black with Vim or Neovim is quick and painless, adding a simple line in your .vimrc or init.vim file. You can also proceed to do a manual installation without a plugin manager as explained in the GitHub repository.

With vim-plug:

 vim
 
Plug 'python/black'

With Vundle:

 vim
 
Plugin 'python/black'

Once the plugin is installed, it may well be convenient to automatically format Python files when saving the buffer, which can be accomplished by adding the following line to the configuration file:

 vim
 
autocmd BufWritePre *.py execute ':Black'

By default, Black works surprisingly well on its own. I only went ahead and changed the default line length from 88 to 79 characters by also adding this line:

 vim
 
let g:black_linelength = 79 " default is 88

Finally, it’s worth noting you can update Black at any point from the terminal with pip and you can also do it directly from within Vim/Neovim with this command:

 vim
 
:BlackUpgrade

Vim will then output something along those lines:

Upgrading Black with pip…

DONE! You are all set, thanks for waiting

Any example?

Taking a random Python 3.6+ code sample, we could originally have something like this (hopefully not):

 python
 
 
 
 for source in data_sources :


 # Retrieve a list of all matching log files in `source`
 log_files=glob.glob( f'{source}/{LOG_NAME}*' )
 if log_files == [ ]:
 print(f'\nThere is no log file to delete in {source}.')
 continue
 else :
 print(f'Log files in {source}:')
 for log_file in log_files:
 if (ARGUMENTS.remind): REMINDER_IS_SET = True
 print(
 log_file
 )
 message = (
 '\nDo you want to delete log files ' 'for this source? (y/n) '
 )
 if (user_says_yes( message = message )) :
 for log_file in log_files: os.remove(log_file)
 print(
 'Log files deleted.'
 )
 continue
 else : continue
 print( 'Exiting script...' )
 sys. exit(0)

Running Black, we will get something that’s quite a bit more palatable:

 python
 
 
 
 for source in data_sources:

 # Retrieve a list of all matching log files in `source`
 log_files = glob.glob(f"{source}/{LOG_NAME}*")
 if log_files == []:
 print(f"\nThere is no log file to delete in {source}.")
 continue
 else:
 print(f"Log files in {source}:")
 for log_file in log_files:
 if ARGUMENTS.remind:
 REMINDER_IS_SET = True
 print(log_file)
 message = (
 "\nDo you want to delete log files " "for this source? (y/n) "
 )
 if user_says_yes(message=message):
 for log_file in log_files:
 os.remove(log_file)
 print("Log files deleted.")
 continue
 else:
 continue
 print("Exiting script...")
 sys.exit(0)

Conclusion

I think the results speak for themselves: enforcing a style with a tool like Black is definitely useful to avoid arguments about conflicting coding styles. It lets you focus on what matters instead: being productive while providing value to the world.

Exponentiation in Modular Arithmetic Made Easy

Sébastien Lavoie — Sat, 27 Apr 2019 19:52:00 -0500

Introduction

The following Python script will by no means provide any useful idea to compete against the RSA algorithm , but it does give you an idea of how a simple technique about reducing the size of an exponent in modular arithmetic can bring you closer to using much larger numbers than the ones you could normally use in the Python interpreter.

The script in action

 python
 
 
 
"""Compute the result of a^b (mod k) by using the exponentiation technique.
The goal here is not efficiency, even though the program is actually pretty
fast: the algorithm is applied manually for demonstration purposes.
Testing on a modest Intel Core i5, having `a` and `b` each set to a random
number containing 2,000 digits and `k` set to a modulo of a number containing
20 digits, results are printed in about 3.3 seconds."""

def binary_remainders(num_b):
 """Take `b` and return the binary equivalent in a list of remainders."""
 remainders = []
 quotient = num_b
 while True:
 prev_quotient = quotient
 quotient //= 2
 remainder = prev_quotient % 2
 remainders.append(remainder)

 if quotient == 0:
 break

 return remainders


def powers_of_two(remainders=None):
 """Return a list of the value of powers of two that form the
 exponent`b`."""
 if remainders is None:
 return None

 powers = []
 for index, remainder in enumerate(remainders):
 if remainder == 1:
 powers.append(2**index)

 return powers


def compute_intermediate_congruences(num_a, num_k, powers=None):
 """Compute all necessary intermediate results of congruence in `mod k` for
 powers of 2 in `powers` to form the number `b`."""
 if powers is None:
 return None

 go_up_to = max(powers)

 intermediate_results = {1: num_a} # Build dictionary to store all results
 start_value = 2 # First power of two to calculate congruence
 congruence = num_a
 while start_value <= go_up_to:
 # value to use for next power of 2
 congruence = congruence ** 2 % num_k
 intermediate_results[start_value] = congruence
 start_value *= 2

 return intermediate_results


def compute_final_congruence(num_k, powers=None, intermediate_results=None):
 """Take all relevant values from `intermediate_results` matching powers in
 `powers`, multiply them together and calculate this number `mod k` to get
 the final result."""
 if intermediate_results is None or powers is None:
 return None

 # store all required values from `intermediate_results`
 congruent_results = []
 for power in powers:
 for key, value in intermediate_results.items():
 if key == power:
 congruent_results.append(value)

 total = 1
 for result in congruent_results:
 total = result * total # Multiply all results together

 return total % num_k # final congruence we are looking for


def compute_congruence(num_a, num_b, num_k):
 """Return `c`, the result of `a^b (mod k)`."""
 num_a = num_a % num_k # Make sure `a` is smaller than `k`

 # Reduce `b` to list of remainders in binary
 remainders = binary_remainders(num_b)

 # Build a list of the powers of 2 forming `b`
 powers = powers_of_two(remainders)

 # Build a list of necessary intermediate results to reach
 # the value of `b` from powers of 2: finds congruence for
 # smaller powers of 2 and store them in a list.
 intermediate_results = compute_intermediate_congruences(num_a, num_k,
 powers)

 # Multiply all relevant intermediate results `mod k` to get the final
 # congruence of `a^b (mod k)`.
 return compute_final_congruence(num_k, powers, intermediate_results)


if __name__ == '__main__':
 print("We will calculate a^b (mod k). Enter only integers.")
 NUM_A = int(input("Provide `a`: "))
 NUM_B = int(input("Provide `b`: "))
 NUM_K = int(input("Provide `k`: "))

 FINAL_RESULT = compute_congruence(NUM_A, NUM_B, NUM_K)
 print(FINAL_RESULT)

Conclusion

It was very interesting to see how a technique that’s applied manually will work wonders with such large numbers. The Python interpreter can barely calculate numbers with exponents with seven digits or more, while a basic approach like the one shown above can quickly churn out the results for seemingly quite large numbers.

As always, this code is available on GitHub.

Have You Been Pwned?

Sébastien Lavoie — Thu, 21 Mar 2019 13:42:00 -0600

Introduction

Have I Been Pwned? has a very simple and accessible API that’s perfect to work with as a beginner. There are different ways to retrieve information and the script below is only one glimpse into the numerous possibilities available. It is built in a way that makes it easy to add features and functionality thanks to Click, a wonderful “command line interface toolkit”.

Python to the rescue

The following script uses Click to build a tiny command-line interface instead of more traditional tools like argparse. You will also need requests to peek inside the API. Both can be obtained as follow in a terminal:

bash

$ pip install click requests

And now, the pièce de résistance.

 python
 
 
 
"""
Simple script that will take advantage of haveibeenpwned.com's API to
find out if your password has been breached.
"""
# Standard library imports
import hashlib

# Third party library imports
import click
import requests


# Needed to access API
PASS_URL = 'https://api.pwnedpasswords.com/range/'

# Script information
VERSION = '0.1.1'
LOGO = r'''
 __ __ __
| |--.---.-.--.--.-----. |__| | |--.-----.-----.-----.
| | _ | | | -__| | | | _ | -__| -__| |
|__|__|___._|\___/|_____| |__| |_____|_____|_____|__|__|

 __ _____
.-----.--.--.--.-----.-----.--| ||__ |
| _ | | | | | -__| _ |', ,-'
| __|________|__|__|_____|_____| |--|
|__| '--' '''


def print_info(context, param, value): # `param` required by `click`
 """Print information about the program and exit."""
 if not value or context.resilient_parsing:
 return
 click.secho(f'{LOGO}\n{"Python 3.6+ Checker".rjust(60)}\n'
 f'{("v" + VERSION).rjust(60)}', fg='blue', bold=True)
 context.exit()


def pass_to_sha1(password):
 """Will convert `password` with SHA-1 algorithm and return the short
 version with the first five characters and the long version with
 everything but the first five characters as a tuple."""

 my_password = password.encode() # converts to byte string

 complete_hash = hashlib.sha1(my_password).hexdigest().upper()
 long_hash = complete_hash[5:]
 short_hash = complete_hash[:5]

 return short_hash, long_hash


def check_password(short_hash, long_hash):
 """Consult haveibeenpwned.com to see how many times the password has
 been breached."""
 click.secho('Please note that only the 5 first characters from your '
 'SHA-1 ENCRYPTED password are sent to haveibeenpwned.com.',
 fg='yellow', bold=True)
 response = requests.get(PASS_URL + short_hash)
 lines = response.text.split()
 for line in lines:
 if long_hash in line:
 num_times = 'occurrences' if int(line[36:]) > 1 else 'occurence'
 click.secho('Your password was found!', fg='red', bold=True)
 click.echo(f'→ {line[36:]} {num_times}')
 break
 else:
 click.secho('Your password was NOT found!', fg='green')


@click.group()
@click.option(
 '--info',
 is_flag=True,
 callback=print_info,
 expose_value=False,
 is_eager=True,
 help="Print some information about the program and exit.")
@click.version_option(version=VERSION, message="%(version)s")
def hibp():
 """A simple command-line interface to make use of
 haveibeenpwned.com's data.

 Type `python3 pwned.py usage` for more info (or whatever way you
 call Python 3)."""


@hibp.command()
def usage():
 """Give examples on how to use this script."""
 click.secho('Examples:', fg='blue')
 click.secho('python3 pwned.py check --help for more info.', fg='green')
 click.secho('python3 pwned.py check -p MyPasswordHere', fg='yellow')
 click.secho('python3 pwned.py check --password MyPasswordHere', fg='green')

 click.echo('')
 click.secho('For more complicated passwords, you have to use quotes and ',
 fg='blue')
 click.secho('escape symbols with \\ where appropriate:', fg='blue')
 click.secho('python3 pwned.py check -p "as0d9\\"asg0\'\'A=)SYD"',
 fg='green')


@hibp.command()
@click.option(
 '-p',
 '--password',
 default=None,
 help="Reveal if match exists with `password`.")
def check(password):
 """Tell if your password has been breached."""
 if password:
 short_hash, long_hash = pass_to_sha1(password)
 check_password(short_hash, long_hash)
 if not password:
 click.secho('Run `pwned.py check --help` for more info.', fg='red')


if __name__ == '__main__':
 hibp()

Example usage

Conclusion

If you are familiar with Python, you can easily make sense of how this little tool works by reading its code. You can also have a look at this repository on GitHub if you want to fork it or reuse it any other way you wish. I have personally found it much easier to work with Click instead of argparse and optparse. I haven’t got to try docopt yet, but it is almost equally as popular as Click on GitHub at the time of this writing and its Pythonic way makes it another great option to consider.

Number of Paths in a Grid… or in Life

Sébastien Lavoie — Thu, 14 Mar 2019 20:38:00 -0600

Introduction

How would you go about determining the number of possible paths in a grid of size n if your initial position is at the top left corner and your destination is the bottom right corner, knowing you can only go right or down at each move? Let’s look at a grid of size 3 × 3:

Even at such a small size, one needs some concentration. Sure, it can be done on paper, but there must be a better way…

Python to the rescue

Once again, programming comes in handy! Life is complicated, but there are aspects of it that can be solved more easily than previously thought when we use the right tools. Knowing that this particular situation corresponds to values from Pascal’s Triangle might help, but it sure is more fun to come up with a different solution. Here is how it occurred to me in Python:

 python
 
 
 
'''
This approach consists in getting the number of possible paths for each
position where we stop in a grid. We know that once we reach the bottom
or the right of the grid, there is only one path possible from there,
so we take advantage of this fact and calculate the total number of
possible paths starting from the bottom right corner. A 2x2 grid would
have 9 indices starting at 0:
012
345
678

We start at index 8 and will calculate the number of possible paths for
all previous positions. From index 8, we calculate that index 4 must
have 2 possible paths to reach index 8 (4>7>8 and 4>5>8). We then find
that index 3 has 3 possible paths since from there we can go right to
index 4 (2 paths) or down to index 6 (1 path). We continue by checking
index 1, which would be the sum of the number of paths at index 4 and
index 2, which is 3 possible paths from there. Now, we are left at the
beginning and the total number of possible paths is index 3 + index 1
(3 + 3 = 6).

The values used would then be tweaked a little depending on the size
of the grid, but the algorithm remains the same. Here is how it works
concretely:

- Get the number of positions in the grid. 2x2 means 9 positions by
counting all intersections. With a 2x2 starting at index 0, we have the
following positions:
012
345
678

- Generate a list corresponding to the length of previously found number
 of positions and give a value of 1 to each position. For a 2x2, it is
 [1,1,1,1,1,1,1,1,1]
- Determine the multiple to be used to check if an index falls in the
 first column (grid_size + 1). The indices in the first column are 0, 3
 and 6, so multiple = 3.
- Start from the bottom right corner of the grid (index 8). Get the
 index of that position starting at zero (length of positions - 1).
- Start a while loop:
 - Check if current position is equal to grid size, meaning we got to
 the last position to evaluate. Return the value of the first index if
 this is the case.
 - Check if current position is a multiple of 3 in that example.
 If so, we skip and go back one index. For instance, if index was 6
 (multiple of 3), go back to index 5.
 - Determine the index of position to the left of current position
 and the index of position above current position. Sum the two together
 and give that value to the index that corresponds to one step left
 and one step up in the grid. Starting at 8, we sum index 7 and index
 5 (result = 2) and set that value for index 4, which is the number of
 possible paths from that index number.
 - Decrement current position by one and repeat the process until we
 break out of the loop.
'''

GRID_SIZE = 100


def num_paths(grid_size):
 '''Return the number of possible paths in a grid of size `grid_size`
 when going only right and down as an integer.'''
 num_points = (grid_size + 1) ** 2
 path_list = [1] * num_points
 multiple = grid_size + 1 # Indices of first column match this multiple

 # Last index in list
 starting_point = len(path_list) - 1

 while True:
 if starting_point == grid_size:
 return path_list[0]
 if starting_point % multiple == 0:
 starting_point -= 1
 else:
 previous = starting_point - 1
 previous2 = starting_point - multiple
 left_up_index = previous2 - 1
 left_up_value = sum([path_list[previous], path_list[previous2]])
 path_list[left_up_index] = left_up_value
 starting_point -= 1


if __name__ == '__main__':
 print(num_paths(GRID_SIZE))

In the end, the idea was to find the number of possible paths that correspond to each possible move, starting from the destination and reverse engineering the number of possibilities until we get to the starting point.

Conclusion

As it turns out, this strategy of reverse engineering things works pretty well when applied to real-life examples. When setting a goal, for instance loosing 10 kilograms in one year, it can sound a lot less daunting to find out what this represents by month (0.83 kg) or even by week (0.19 kg) instead of fixating on the big gap from your current situation. For short-term goals, this can be reduced to daily or even hourly outcomes.

This goes hand in hand with the advice found in Atomic Habits, an excellent book written by James Clear:

A 1% daily improvement leads to a version of yourself that’s 37 times better in one year.

If that’s not motivating enough to start taking little steps with consistency towards your dreams, I don’t know what is.

Executing Shell Commands in a Python Script

Sébastien Lavoie — Fri, 08 Mar 2019 14:34:00 -0600

Introduction

Python is awesome for producing high-quality code quickly and efficiently, but it is not necessary to reinvent the wheel in each project: this is why the third-party library on PyPI is so extensive. There are also occasions when it is convenient to execute shell commands available in the terminal, either to retrieve its output or to perform some work in the background. I have wanted to do both when building this website, so here is one way to accomplish this!

Retrieve the output

 python
 
 
 
def read_tree():
 '''Execute `tree` command and store the output in
 `tree.txt`.'''
 tree = subprocess.getoutput(f'tree .')
 print(tree)
 with open('./tree.txt', 'w') as f:
 f.write(tree)


read_tree()

In this example, the structure from the current directory is printed when executing the code and it is later stored in a file. The output could look something like the following:

 
 
 
.
├── database_example
│   ├── example.sqlite
│   ├── db_manager.py
│   ├── review
│   │   └── sql_queries.txt
│   └── settings.py
├── database_sql.py
├── dump.sql
├── example.db
├── example.db.sql
├── example_info.txt
├── example_old.db
├── example_old.db.sql
├── sample
│   ├── chinook.db
│   └── sqlite-sample-database-diagram-color.pdf
├── test.db
└── test.sql

3 directories, 15 files

Execute a command in the background

There are also instances in which is it useful to execute a program from the terminal inside a Python script. This website, for example, currently exports daily Git statistics in the output/ folder. This looks like this:

 python
 
 
 
def daily_stats():
 '''Execute `Gitstats` once a day based on the date found in
 `stats_counter.txt`. Very simple with a caveat: it won't check if
 there are new commits on the same day if stats have already been
 generated on that day.'''
 today = datetime.today().strftime('%Y%m%d')
 with open('stats_counter.txt') as f:
 content = f.readline().strip()
 if content != today:
 current_loc = current_path()
 cmd = ["gitstats", "-c", "project_name='sglavoie.com'",
 f"{current_loc}", f"{current_loc}/output/stats/"]
 subprocess.run(cmd)
 with open('stats_counter.txt', 'w') as f:
 f.write(today)

This is an automated process, which is something Python is easy to use for. However, it won’t be necessary to program everything: it all gets generated on demand!

Conclusion

I am always amazed at how easy it can be to automate simple tasks like those mentioned above. Programming can be very rewarding regardless of initial abilities. Just beyond the most basic stuff, suddenly there is a world that opens up to those that stay curious long enough.

Binary to Decimal, Decimal to Binary Converter

Sébastien Lavoie — Mon, 18 Feb 2019 21:30:00 -0600

Introduction

There is nothing new here in terms of algorithms, but it was interesting to figure out how to apply those concepts with Python. This works for integer numbers, but can be easily adapted to convert fractions.

Binary to Decimal

 python
 
 
 
"""
Converts a binary number to a decimal number.
"""


def bin_to_dec(number):
 """Algorithm that converts a binary number to a decimal number.

 Receives and returns a string."""

 # get the index of power of first digit
 # e.g. 1001 → first index power is 3 (2^3)
 first_power = len(number) - 1

 total = 0 # initialize variable to store final result

 # iterate over each number and if it is '1', calculate the value
 # based on its position and add it to the total
 for index, char in enumerate(number):
 if char == '1':
 total += 2 ** (first_power - index)

 # give back the answer to the script
 return total


if __name__ == '__main__':
 # retrieve binary number from input as string to parse
 BIN_NUM = input()

 # pass binary number to algorithm and store the result
 RESULT = bin_to_dec(BIN_NUM)

 # print the solution to the console
 print(RESULT)

Decimal to Binary

 python
 
 
 
"""
Converts a decimal number to a binary number.
"""


def dec_to_bin(number):
 """Algorithm that converts a decimal number to a binary number.

 Receives an integer and returns a string."""

 if number == 0: # if number is 'zero', the answer is 'zero'
 return 0

 remainders = [] # list that stores the remainders

 # iterate over the initial given decimal and divide it by two
 # until it gets to 'zero'. At each step, add the remainder in
 # the above list
 while number > 0:
 remainders.append(number % 2)
 number = number // 2

 # reverse the list of remainders, as the first remainder is the
 # right most digit in the answer
 remainders.reverse()

 # concatenate all the digits from the list of remainders from left
 # to right to display the final binary number
 answer = ''
 for digit in remainders:
 answer += str(digit)

 # give the answer back to the script
 return answer


if __name__ == '__main__':
 # retrieve decimal number from input as integer
 DEC_NUM = int(input())

 # pass decimal number to algorithm and store the result
 RESULT = dec_to_bin(DEC_NUM)

 # print the solution to the console
 print(RESULT)

Conclusion

This was a very quick programming session that’s been useful to make new concepts stick. The best learning strategies are to test yourself constantly in any way possible and to teach clearly what you know, which is what I tried to do in the end. By the way, I found the course Learning How to Learn on Coursera to be quite satisfying and I would invite you to have a look if that’s a topic you’re interested in!

Bash History Cleaner

Sébastien Lavoie — Sun, 23 Dec 2018 19:35:00 -0600

Introduction

This is a Python 3.6+ script that helps to clean the file containing the Bash history commands. It will remove any line matching a specified regular expression and can also remove any line starting with an alias.

The idea behind this small utility was simple:

The Bash history file (usually located in ~/.bash_history) contains much of the work one ends up doing in the terminal.
The history can grow large over time and it becomes more cumbersome to find interesting information in all that clutter, such as a rarely used command with specific flags.
By removing all superfluous commands that are repeated often and which give no real benefit in certain contexts (such as ls, cd, cat, etc.), the history is much cleaner and easier to navigate and actually becomes much more useful in my opinion.
True, it will be harder to follow the bread crumbs for everything you did, but I haven’t come across a situation where having access to yet another empty ls or cd has proven necessary and reading .bash_history doesn’t make for a great narrative story either.

Make history in a big way

I took advantage of the fact that the history can be cleaned with the script you are about to see and set up what is known as an eternal history which, as it sounds like, can grow infinitely big! All you have to do is append the following lines to the file ~/.bashrc:

 bash
 
 
 
# Eternal bash history.
# ---------------------
# Undocumented feature which sets the size to "unlimited".
# http://stackoverflow.com/questions/9457233/unlimited-bash-history
export HISTFILESIZE=-1
export HISTSIZE=-1
export HISTTIMEFORMAT="[%F %T] "
# Change the file location because certain bash sessions truncate
# .bash_history file upon close.
# http://superuser.com/questions/575479/bash-history-truncated-to-500-lines-on-each-login
export HISTFILE=~/.bash_eternal_history
# Force prompt to write history after every command.
# http://superuser.com/questions/20900/bash-history-loss
PROMPT_COMMAND="history -a; $PROMPT_COMMAND"

Bash History Cleaner

And here is the script in question *. It comes in two files that need to be in the same directory:

One is a Python file that needs to be launched from the terminal with Python 3.
The other file, settings.json, is a JSON file used to store the settings of the script, which will be detailed below.

* _{Improvements to the original script can be found on Github. To keep this
article a bit more readable, the original version is shown.}

`bash_history_cleaner.py`

 python
 
 
 
'''
Python script that helps to clean the file containing the Bash history
commands.

Note: Requires Python 3.6+

It will remove any line matching a specified regular expression and can
also remove any line starting with an alias.

Description of available settings in `settings.json`:

 "home_directory": Absolute path to user's home directory.

 "history_file": Name of file where the history will be cleaned up.

 "aliases_file": Name of file where Bash aliases are set up.

 "ignore_patterns": List of patterns to ignore in `history_file`.
 Each line where a pattern is found will be deleted.
 → Patterns are specified as regular expressions.

 "add_aliases": Boolean. If set to `true`, aliases from `aliases_file`
 will be added to `ignore_patterns`.

 "aliases_match_greedily":
 Boolean. If set to `true`, any line in `history_file`
 starting with an alias in `aliases_file` will be
 deleted. If set to `false`, delete line if the alias is
 the content of the whole line (with optional space at
 the end): `false` matches "^alias$" or "^alias $" only.

 "backup_history": Boolean. If set to `true`, `history_file` will be backed
 up in the same directory with a name ending in .bak
 based on the current date.

 "delete_logs_without_confirming":
 Boolean. If set to `true`, script with flag `-c` will
 automatically delete all the backup files found for
 `history_file`.
'''
from datetime import datetime
from pathlib import Path
import argparse
import fileinput
import glob
import json
import os
import re


def get_current_path():
 '''Returns the current working directory relative to where this script
 is being executed.'''
 return Path(__file__).parents[0]


def user_says_yes(message=""):
 '''Check if user input is either 'y' or 'n'. Returns a boolean.'''

 while True:
 choice = input(message).lower()
 if choice == 'y':
 choice = True
 break
 elif choice == 'n':
 choice = False
 break
 else:
 print("Please enter either 'y' or 'n'.")

 return choice


def delete_logs(settings: dict, history_file: str):
 '''Delete log files in `home_directory` based on `history_file`.'''

 # Retrieve a list of all matching log files
 log_files = glob.glob(f'{history_file}_*.bak')

 if log_files == []:
 print("There is no log file to delete.")
 else:
 print(f'Log files found in {settings["home_directory"]}:')

 for log_file in log_files:
 print(log_file)

 if settings['delete_logs_without_confirming']:
 for log_file in log_files:
 os.remove(log_file)
 print('Log files deleted.')
 return

 message = ("\nDo you want to delete those log files? [y/n] ")

 if user_says_yes(message=message):
 for log_file in log_files:
 os.remove(log_file)
 print('Log files deleted.')
 return

 print('Operation aborted.')
 return


def generate_date_string() -> str:
 '''Return date formatted string to backup a file.'''
 return datetime.strftime(datetime.today(), '_%Y%m%d_%H%M%S.bak')


def load_settings(settings_file: str) -> dict:
 '''Load settings in the script. Return them as a dictionary.'''
 with open(settings_file, "r") as read_file:
 settings = json.load(read_file)
 return settings


def get_list_aliases(bash_aliases_file: str, settings: dict) -> list:
 '''Retrieve the name of all the aliases specified in `bash_aliases_file`.

 Return aliases as a list of strings formatted as regular expressions.'''

 match_whole_line = bool(settings['aliases_match_greedily'])

 with open(bash_aliases_file) as file:
 content = file.read().splitlines() # one alias per line
 aliases_list = []
 for line in content:
 try:
 # Get the actual alias in each line
 # Use negative lookbehind to remove 'alias ' at the beginning.
 # Matches anything after that is a dot, digit, underscore
 # or letter (will stop at the equal sign: alias blah='...')
 alias = re.search(r'(?<!not )alias ([\.\w]*)', line).group(1)

 # If for some reason the alias cannot be extracted, skip it
 # If search doesn't match, it's of type None and won't work
 except AttributeError:
 continue

 if match_whole_line:
 # Match the whole line if it starts with the alias.
 alias = f'^{alias}( )?$|^{alias} .*'
 else:
 # Will match only when alias is the whole content of the line,
 # followed by optional space.
 alias = f'^{alias}( )?$'

 # Escape dots in alias
 alias = alias.translate(str.maketrans({".": r"\."}))

 aliases_list.append(alias)

 return aliases_list


def clean_bash_history(settings: dict, history_file: str):
 '''Modify in place `history_file` by removing every line where
 `ignore_patterns` is found.

 Optionally, add a list of aliases to `ignore_patterns` with
 `aliases` based on the value of `add_aliases` in settings.json.'''

 if settings['backup_history']:
 backup_str = generate_date_string()
 file_input = fileinput.FileInput(history_file,
 inplace=True,
 backup=backup_str)
 else:
 file_input = fileinput.FileInput(history_file,
 inplace=True)

 with file_input as file:
 for line in file:
 has_match = False
 for pattern in settings['ignore_patterns']:
 matched = re.compile(pattern).search
 if matched(line):
 has_match = True
 break
 # If no match is found (nothing to ignore), print the line
 # back into the file. Otherwise, it will be empty.
 if not has_match:
 print(line, end='') # Line already has carriage return


def launch_cleanup(settings: dict, history_file: str, aliases_file: str):
 '''Main function that launches the cleanup process.'''

 bash_aliases = None
 if settings['add_aliases']:
 try:
 bash_aliases = get_list_aliases(aliases_file, settings)

 # add aliases to list of patterns to ignore
 settings['ignore_patterns'].extend(bash_aliases)
 except FileNotFoundError:
 print(f"File not found: {aliases_file}")
 quit()
 try:
 clean_bash_history(settings, history_file)
 except FileNotFoundError:
 print(f"File not found: {history_file}")


if __name__ == '__main__':
 SETTINGS_FILE_PATH = get_current_path() / 'settings.json'
 SETTINGS = load_settings(SETTINGS_FILE_PATH)
 ALIASES_FILE = SETTINGS['home_directory'] + '/' + SETTINGS['aliases_file']
 HISTORY_FILE = SETTINGS['home_directory'] + '/' + SETTINGS['history_file']

 # initiate the parser to check all the arguments passed to the script
 PARSER = argparse.ArgumentParser()
 PARSER.add_argument(
 '-c', '--clear', help='Delete all log files', action='store_true')

 # read arguments from the command line
 ARGUMENTS = PARSER.parse_args()

 if ARGUMENTS.clear:
 delete_logs(SETTINGS, HISTORY_FILE)
 quit()

 launch_cleanup(SETTINGS, HISTORY_FILE, ALIASES_FILE)

`settings.json`

 json
 
 
 
{
 "home_directory": "/home/sglavoie",
 "history_file": ".bash_eternal_history",
 "add_aliases": true,
 "aliases_file": ".bash_aliases",
 "aliases_match_greedily": true,
 "backup_history": true,
 "delete_logs_without_confirming": false,
 "ignore_patterns": [
 "^\\#\\d+",
 "^$",
 "^(\\.\\/)?pip$",
 "^(\\.\\/)?python.*$",
 "^\\.\\.$",
 "^alias",
 "^cd ",
 "^cd$",
 "^cd..$",
 "^fg$",
 "^df( )?",
 "^du( )?",
 "^exit$",
 "^git branch",
 "^git checkout master",
 "^git log$",
 "^git push$",
 "^git status$",
 "^git stauts$",
 "^kill \\d+.*",
 "^ls -a$",
 "^ls$",
 "^make|make install$",
 "^man ",
 "^pelican( )?$",
 "^pip install -r requirements.txt",
 "^pip.* list|pip.* show$",
 "^source ",
 "^sudo apt-get autoclean|sudo apt-get autoremove$",
 "^sudo apt-get dist-upgrade|sudo apt-get update$",
 "^which "
 ]
}

Description of available settings in `settings.json`

Name of setting	Description
`home_directory`	Absolute path to user’s home directory.
`history_file`	Name of file where the history will be cleaned up.
`aliases_file`	Name of file where Bash aliases are set up.
`ignore_patterns`	List of patterns to ignore in `history_file`. Each line where a pattern is found will be deleted. Patterns are specified as regular expressions.
`add_aliases`	Boolean. If set to `true`, aliases from `aliases_file` will be added to `ignore_patterns`. (Default: `true`)
`aliases_match_greedily`	Boolean. If set to `true`, any line in `history_file` starting with an alias in `aliases_file` will be deleted. If set to `false`, delete line if the alias is the content of the whole line (with optional space at the end): `false` matches “^alias$” or “^alias $” only.
`backup_history`	Boolean. If set to `true`, `history_file` will be backed up in the same directory with a name ending in .bak based on the current date. (Default: `true`)
`delete_logs_without_confirming`	Boolean. If set to `true`, script with flag `-c` will automatically delete all the backup files found for `history_file`. (Default: `false`)

Anecdotal evidence of satisfying performances

Performance-wise, this scans ~8,300 lines per second on my modest Intel Core i5 laptop with files of over 200,000 lines long. Not that I type so much stuff in the terminal: I just duplicated many lines.

Conclusion

This is a simple solution to an nonexistent problem, but it was in the end very instructive to me nonetheless. You may even find a use for it! Otherwise, you might use the same functions for other files such as logs! If you would like to take a closer look at the source in a more convenient way, you can find the code available on Github.

sglavoie.com – Tag: Python

From a plan to unattended agent runs: how I drive coding agents one session at a time

Introduction

Why plans, not prompts

The pipeline, step by step

Splitting: where to cut

Expanding: the plan is a set of claims

Preflight: the gate

Completion contracts: who decides what “done” means

What a run looks like

Fresh context, durable handoffs

Pausing is a feature

The controls I use all the time

A second agent, reviewing cold

Guardrails, briefly

When this is overkill

Lessons from four months of building it

Conclusion

Resources and references

External links

From this website

Building the next Facebook in a couple of weeks (as a humble experiment)

Introduction

Meeting the requirements

Running the application

Installing and running the web application

Accessing user accounts

Running locally

Backend

Frontend

Using Docker

Local deployment

Summary of the steps needed to deploy locally

Functionality implemented

Executing the unit tests

Logical approach

Frontend framework: React

React components and icons: Material UI

API: Django Rest Framework

User management: Djoser

Asynchronous communication

Deployment: Docker, Cloudflare Tunnel

Formatting: Prettier, Black, Vulture, Pylint

API calls: Axios

State management: Redux and Immer

Design and implementation decisions

Evaluation of the final product

What worked well

What could have been improved

State of the art in web development

Demo of the app

Resources and references

Using Google Sheets as a database to extract data converted to Markdown

Introduction

General software requirements

Input

Output

Installing this tool

Using pip

Using pipenv

How to use this tool

Conclusion

Convenient and lightning fast backups with rsync

Introduction

Prerequisites

Setting up the script

The configuration file

Description of available settings

The script

How to use

One-time setup

Using as a daily driver

Avoid deleting files at the destination

Conclusion

More resources and references

Black: A Python Formatter to Brighten Your Workflow

What is Black?

How to use it?

Use it from the terminal

Use it with Vim

Using `pip`

Using `pipenv`

`bash_history_cleaner.py`

`settings.json`

Description of available settings in `settings.json`