sglavoie.com – Tag: Shell

Ignoring Sync of Local Files to Dropbox on Linux

Sébastien Lavoie — Sat, 30 Nov 2019 13:22:00 -0600

Introduction

Dropbox is an awesome company which has been until recently the working home of none other than Guido van Rossum, the creator of the Python programming language. It works like a charm out of the box on Linux and can be installed easily enough through the package manager that comes with your chosen distribution.

Why Dropbox?

In my experience, it has been working flawlessly and syncs consistently faster than Google Drive. It’s also worth mentioning that there is still no official Google Drive Linux client to this day: some users of the GNOME desktop environment will connect to their Google account and mount Google Drive within the file manager Nautilus/Files; other users will use the paid software Insync or the free software alternative ODrive. No thanks.

On the other hand, Dropbox is easy to use with any flavor of Linux and also syncs seamlessly with mobile devices. I use Dropbox for its convenience, but other solutions exist like Syncthing that do not depend on proprietary software and third-party cloud services. I store mostly insensitive information in my Dropbox account and for the few files that happen to be a bit more sensitive (such as the password database I synchronize on both desktop and mobile with KeePassXC and Keepass2Android, respectively), they are of course encrypted (hopefully in a secure way).

Because I like to keep things simple and light on system resources, I do not let Dropbox run in the background (it only syncs once a day in the evening). Since all one needs to do in order to use it is to create and modify files inside a specific folder (such as ~/Dropbox by default), it can just synchronize at any moment and conflicts will be handled automatically if you make changes in more than one place that’s being synced.

So, what’s wrong with Dropbox?

With all that said, there’s one little feature that I find missing with Dropbox when using the official Linux client and that’s the ability to selectively sync local files. Yes, there is a selective sync feature available, but that will delete the local copy of your files and store them in the cloud only, not the other way around by keeping your local files intact and prevent uploading them to the cloud. There are some hacks you can do to avoid this, but this isn’t a particularly enticing work flow, because we can do better.

You may have heard of other alternatives like GoodSync that allow, through yet another third-party proprietary software, to ignore specific files at your request. That’s not what we are after here: instead, there’s something you can do from the terminal that works quite well, too. At the time of this writing, it is still a beta feature according to this page:

This feature is currently in beta and not available to all Dropbox users. It will be rolled out to more users in the future.

Yet, because you can work with this feature programmatically to ignore and un-ignore files from your Dropbox directory, I thought it was worth giving it a try. Here is how I’m using it to give you a few ideas.

Any simple example?

Getting started is quick and painless. To ignore a file within your Dropbox folder, you only need to use the attr (on Linux, else xattr on macOS) command to set (-s) (or -w write on macOS) the special attribute com.dropbox.ignored to give it a value of 1 (-V 1) so it ignores a file (file.txt in this example) as follow:

 bash
 
cd /path/to/file/in/Dropbox/folder

attr -s com.dropbox.ignored -V 1 file.txt # Linux
xattr -w com.dropbox.ignored 1 file.txt # macOS

Ignore all .git and node_modules folders

Things get really interesting when we can do that automatically by matching a specific pattern (it depends who you ask, anyways). How about exploring recursively the whole directory tree to exclude only hidden .git folders? Suppose you store your Git projects in ~/Dropbox/git/. You would then do something like this:

 bash
 
 
 
cd ~/Dropbox/git/

# Linux
find . -type d -name ".git" |
 xargs -I {} attr -s com.dropbox.ignored -V 1 "{}"

# macOS
find . -type d -name ".git" |
 xargs -I {} xattr -w com.dropbox.ignored 1 "{}"

In an instant, you will start getting some feedback printed to the terminal for each .git folder that is being found:

Attribute "com.dropbox.ignored" set to a 1 byte value for ./path/to/.git:
1

Dealing with node_modules folders is equally easy. Because a message will be printed to the console every time a file attribute is changed, you could want to delete the output (preferably once you are sure you are modifying only the targeted files!) by redirecting it to /dev/null, a null device that discards all data it receives. We can always split a long command onto multiple lines with a backslash character \ like so (it’s not required right after the pipe | symbol):

 bash
 
find . -type d -name "node_modules" |
 xargs -I {} attr -s com.dropbox.ignored -V 1 \
 "{}" > /dev/null

Can we make this simpler?

Yes we can! An alias might be a bit hard to manage, but we can definitely create a set of functions that will be accessible from anywhere in the terminal. It suffices to add them to your shell configuration file, which could be located in ~/.profile, ~/.bash_profile, ~/.zshrc or something else depending on your shell.

One solution that I see fit for this purpose is to have two distinct functions that have antagonistic effects so that we can revert our changes in a pinch. Basically, setting a value of 1 for the attribute com.dropbox.ignored ignores a file while setting it back to 0 tells Dropbox to sync it again. Our functions could be conveniently called dropbox-ignore and dropbox-sync.

Now, the following functions could become quite a bit more complicated to take into account various scenarios and custom behaviors, but let’s keep things simple and assume that the user knows what is going to happen. We could have our two functions as follow:

 bash
 
 
 
# Ignore specific files/directories in Dropbox
dropbox-ignore() {
 arg1=$1
 arg2=$2

 if [ -z "$arg1" ] || [ -z "$arg2" ]; then
 echo "Usage: dropbox-ignore <f|d> <name> (e.g. dropbox-ignore d .git)"
 return 1
 fi

 find ~/Dropbox -type $arg1 -name "$arg2" |
 xargs -I {} xattr -w com.dropbox.ignored 1 "{}"
}

# Sync specific files/directories in Dropbox
# that were previously ignored (or not)
dropbox-sync() {
 arg1=$1
 arg2=$2

 if [ -z "$arg1" ] || [ -z "$arg2" ]; then
 echo "Usage: dropbox-sync <f|d> <name> (e.g. dropbox-sync d .git)"
 return 1
 fi

 find ~/Dropbox -type $arg1 -name "$arg2" |
 xargs -I {} xattr -d com.dropbox.ignored "{}"
}

How do you use those functions?

Our commands will be accessible by typing dropbox-ignore and dropbox-sync the next time we open a terminal window. Both functions do exactly the same thing, but dropbox-ignore sets the attribute value to 1 to ignore files and dropbox-sync sets the attribute to 0 to allow syncing to happen.

_{Note: For simplicity and practicality, keep in mind that those commands will search recursively from the specified directory. It’s also good to know that if you set an attribute value to 1 then back to 0 again, Dropbox will need to re-sync the affected files and directories.}

We need to pass them two arguments: the first is the type of search to perform (pass f for files and d for directories) and the second is the pattern to match in the name (node_modules or .git would fit here, no need to use quotes). Concretely, that will look like this:

 bash
 
# To ignore a file matching the pattern `myfile`
dropbox-ignore f myfile

# To avoid syncing node_modules/ directory
dropbox-ignore d node_modules

# Te re-add a .git/ folder to Dropbox
dropbox-sync d .git

Conclusion

This tip may certainly not be the best approach to solving this particular issue with Dropbox on Unix, but I hope you find it useful nevertheless as it worked out very nicely on my end! One could store files and directories elsewhere, working with symbolic links or hard links as necessary to reference files or simply give up on making a backup with Dropbox in some circumstances. But I like to keep an extra copy of projects I’m working on in Dropbox and that can include potentially large files that don’t need to be backed up, so ignoring what’s not indispensable can speed up the syncing process tremendously while keeping disk usage in the cloud possibly much lower.

Dropbox only calculates the size of the files being synced in the cloud, so you can end up needing a lot more space locally if you ignore a number of big files or, conversely, selectively syncing many files will result in a cloud storage larger than what’s needed locally.

fzf - A Fuzzy Finder to Accomplish Anything

Sébastien Lavoie — Fri, 24 May 2019 10:56:00 -0500

Introduction

I know people who make such a mess when not organizing their files and directories appropriately on their system, they would benefit greatly from fzf.

What is fzf?

From the official GitHub page:

fzf is a general-purpose command-line fuzzy finder. It’s an interactive Unix filter for command-line that can be used with any list; files, command history, processes, hostnames, bookmarks, git commits, etc.

Terminal aliases

One of the main uses of fzf is from the terminal. Being so flexible to use, it can be combined with all kinds of commands with the help of pipes (|) to bend it to your desires. Here are some aliases I am currently using that undoubtedly improve my terminal workflow.

 bash
 
##### Functions

# Select a configuration file with fzf and open it with Neovim
conf() { du -a ~/.dotfiles/* ~/.config/* | awk '{print $2}' | fzf | xargs -r nvim ;}

# Select a file from current folder and recursively with fzf and open it with Neovim
se() { du -a ./* | awk '{print $2}' | fzf | xargs -r nvim ;}

# Select a file recursively from university folder with fzf and open it with default app
sc() { du -a ~/Dropbox/university/* | awk '{print $2}' | fzf | xargs -r xdg-open ;}

The first alias, conf (short for configuration), allows to search only within the two folders specified for configuration files, which makes it pop almost instantaneously since it doesn’t have to scan files scattered anywhere else. You can then type anything that partially matches a file path and even include slashes (/) in your match if you know in which directories to look. You can then type Enter to open the file with your favorite text editor (here set to nvim) or type CTRL + c to abort the command.

The other aliases work in a similar fashion. se (short for search) will simply search recursively for any kind of files in the current directory and open the selected file in a text editor. Of course, certain file types can be excluded and everything else can be tweaked with more piping power. In the case of the third alias, sc (short for school), it will also search recursively for any kind of file in the university folder and will open it with the default application set to open that kind of file (video, image, text, PDF, etc.).

Vim/Neovim integration

Using Vim-Plug from the same author as fzf is as easy as adding the following to Vim/Neovim’s configuration file. First, we make sure that fzf is available on the system and install it if it’s not. Then, we install the fzf.vim plugin to integrate fzf in Vim and optionally, we may configure more options so that fzf can be used more optimally, which can be found on the GitHub plugin page.

 vim
 
 
 
Plug 'junegunn/fzf', { 'dir': $HOME . '/.fzf', 'do': './install --all' }
Plug 'junegunn/fzf.vim'

""""" [ FZF ]
" Allows fzf to ignore patterns in .gitignore
let $FZF_DEFAULT_COMMAND = 'ag -g ""'

" Mapping selecting mappings
nmap <leader><tab> <plug>(fzf-maps-n)
xmap <leader><tab> <plug>(fzf-maps-x)
omap <leader><tab> <plug>(fzf-maps-o)

" Insert mode completion
imap <c-x><c-k> <plug>(fzf-complete-word)
imap <c-x><c-f> <plug>(fzf-complete-path)
imap <c-x><c-j> <plug>(fzf-complete-file-ag)
imap <c-x><c-l> <plug>(fzf-complete-line)

" Advanced customization using autoload functions
" (expand word completing window)
inoremap <expr> <c-x><c-k> fzf#vim#complete#word({'left': '20%'})

" Make use of fzf command instead of CtrlP
map <C-p> :FZF<cr>
""" [ / FZF ]

Working with Neovim

fzf really shines when used with Neovim as it is extremely fast, especially when configured with The Silver Searcher which deserves its own article. It integrates with core functionality of Vim and makes it easy to find what you are looking for.

From the above configuration, for instance:

 vim
 
nmap <leader><tab> <plug>(fzf-maps-n)

This allows to search for existing mappings and commands, which can be faster than diving in the help pages when looking for a quick reference.

This one in insert mode is very handy:

 vim
 
imap <c-x><c-k> <plug>(fzf-complete-word)

Without affecting keyword completion with CTRL + n and CTRL + p (unless you have set complete+=k in your configuration file), you can complete words from a custom dictionary of your choice with CTRL + x CTRL + k. The window that appears can be moved and resized, which is what is happening here:

 vim
 
inoremap <expr> <c-x><c-k> fzf#vim#complete#word({'left': '20%'})

In the same way, file paths can be completed from the current working directory CTRL + x CTRL + f and existing lines can be quickly inserted with CTRL + x CTRL + l.

Finally, I use fzf to open any file quickly from the working directory inside Neovim with the mapping CTRL + p, which replaces the CtrlP plugin and can work much faster on larger codebases.

Conclusion

There is much more that can be done with it and I am barely scratching the surface here. Luke Smith shared a great video that will complement the information from this post nicely with a more technical approach. Highly recommended to see more practical ways to use this superb piece of software!

Luke Smith also makes a strong point of combining fzf with dmenu, a tool that I described succinctly in a previous article.