From 507c7bbc4f74e8ca972d1b248e7469bc4315046d Mon Sep 17 00:00:00 2001 From: Daniel Lange Date: Tue, 23 Feb 2016 08:50:37 +0100 Subject: Initial commit of cleaned up github:sickill/bitpocket repository --- README.md | 231 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 231 insertions(+) create mode 100644 README.md (limited to 'README.md') diff --git a/README.md b/README.md new file mode 100644 index 0000000..052bc85 --- /dev/null +++ b/README.md @@ -0,0 +1,231 @@ +# bitpocket + +[![Build Status](https://secure.travis-ci.org/sickill/bitpocket.png?branch=master)](http://travis-ci.org/sickill/bitpocket) + + +## About + +**bitpocket** is a small but smart script that does 2-way directory +synchronization. It uses _rsync_ to do efficient data transfer and tracks local +file creation/removal to avoid known rsync problem when doing 2-way syncing +with deletion. + +bitpocket can use any server which you have ssh access to for its central +storage. If you have gigabytes of free disk space on your hosting server you +can finally make use of it. + + +## Installation + +Clone repository and symlink `bitpocket` bin to sth in your `$PATH`: + + $ git clone git://github.com/sickill/bitpocket.git + $ ln -s `pwd`/bitpocket/bin/bitpocket ~/bin/bitpocket + +Or download script and place it in a directory in your `$PATH`: + + $ curl -sL https://raw.github.com/sickill/bitpocket/master/bin/bitpocket > ~/bin/bitpocket + $ chmod +x ~/bin/bitpocket + + +### Setting up master + +Create an empty directory on some host that will be the master copy of your files: + + $ ssh user@example.org + $ mkdir ~/BitPocketMaster + + +### Setting up slaves + +On each machine you want to synchronize initialize an empty directory as your bitpocket: + + $ mkdir ~/BitPocket + $ cd ~/BitPocket + $ bitpocket init user@example.org ~/BitPocketMaster + + +## Usage + +After installation, you use the `bitpocket` command for synchronization and other tasks. +Running `bitpocket help` will display the following message. + + usage: bitpocket [sync|help|pack|log|cron|list|init ] + + Available commands: + sync Run the sync process. If no command is specified, sync is run by default. + init Initialize a new bitpocket folder. Requires remote host and path params. + pack Pack any existing (automatic) backups into a git repository. + cron Run sync optimized for cron, logging output to file instead of stdout. + log Display the log generated by the cron command + list List all files in the sync set (honoring include/exclude/filter config). + help Show this message. + + Note: All commands (apart from help), must be run in the root of a + new or existing bitpocket directory structure. + + +### Manual sync (bitpocket sync) + +To synchronize your local slave with master just run _bitpocket sync_ inside your +bitpocket directory: + + $ cd ~/BitPocket + $ bitpocket sync + +Ensure that you run bitpocket at least once immediately after creating a new slave and +before adding new files to the slave directory. If there are files in the master they +will be pulled into the slave. You may then move files into your slave directory and +they will be detected as added. + + +### Maintaining backups (bitpocket pack) + +bitpocket does not include a full-fledged versioning system at the moment, but +it does automatically create a local backup of any files before overwriting or +deleting with changes from the BitPocketMaster. These backups are placed into a +timestamped directory (one directory per sync). The path to this directory is: + + .bitpocket/backups/YYYY-MM-DD.hhmmss + +As these files accumulate, you may want to remove them, but you can also run +`bitpocket pack` to combine all the backup files into a git repository contained +within the _.bitpocket_ directory: + + $ cd ~/BitPocket + $ bitpocket pack + +This requires an installation of _git_, but +allows all the space saving advantages of _git_ when making repeated changes +to the same files. + +There is a discussion about potential directions for versioning direction here: +[github.com/sickill/bitpocket/issues/15](https://github.com/sickill/bitpocket/issues/15) + + +### Redirecting output to log file (bitpocket cron) + +If bitpocket is run with the cron parameter ( _bitpocket cron_ ), it will perform +a sync, but instead of showing the progress on stdout, it will redirect all +output to a log file: + + $ cd ~/BitPocket + $ bitpocket cron + +As the name of this parameter implies, this is mainly useful when running bitpocket +through the _cron_ command. (See "Automatic sync with cron" for more information +about how to configure this). + + +### Displaying logs (bitpocket log) + +When running bitpocket in cron with `bitpocket cron` it will append its output +to _.bitpocket/log_ file. You can review the tail end of an existing log file, +or watch live log as it is generated, with following command: + + $ cd ~/BitPocket + $ bitpocket log + + +### Displaying list of files to be synchronized (bitpocket list) + +You may want to know which files will be synchronized before actually performing +the syncronization. You can verify which files are in the synchronization set +by running `bitpocket list`: + + $ cd ~/BitPocket + $ bitpocket list + +Note that this does _not_ list changed files, it only lists all the local files +that bitpocket will look at in determining which files to sync. Also, note that +if there are new files in the master that will be added on a sync, they will +not be included here. This command is only intended to verify which files are +in the synchronization set. (See "Configuring file exclusion and inclusion" for +information about how to control which files are in the synchronization set). + + +## Configuration + +### Automatic sync with cron + +Add following line to your crontab to synchronize every 5 minutes: + + */5 * * * * cd ~/BitPocket && nice ~/bin/bitpocket cron + +Note that cron usually has very limited environment and your ssh keys with +passphrases won't work in cron jobs as ssh-agents/keyrings don't work there. +Thus it's preferable to generate passphrase-less ssh key for bitpocket +authentication: + + $ cd ~/BitPocket + $ ssh-keygen -t rsa -C bitpocket-`hostname` -N '' -f .bitpocket/id_rsa + $ ssh-copy-id -i .bitpocket/id_rsa user@example.org + +and uncomment line with `RSYNC_SSH` in _.bitpocket/config_ file. + + +### Custom rsync options + +You can pass additional switches to `rsync` by setting `RSYNC_OPTS` in +_.bitpocket/config_ file. Generated config file includes (commented out) +example setting for dereferencing symlinks: + + # RSYNC_OPTS="-L" + +Just uncomment it and change at will. + + +### Configuring file exclusion and inclusion + +If you want some files to be ignored by bitpocket you can create +_.bitpocket/exclude_ file and list the paths there: + + *.avi + jola + /misio.txt + +_*.avi_ and _jola_ will be matched anywhere in path, _misio.txt_ will be +matched at bitpocket root dir ( _~/BitPocket/misio.txt_ ). + +This exclude file is passed to `rsync` as `--exclude-from` argument, check `man +rsync` for _INCLUDE/EXCLUDE PATTERN RULES_. + +You can set up even more advanced exclusion/inclusion rules. In all, there +there are three files that you can create to change this configuration: + + .bitpocket/exclude + .bitpocket/include + .bitpocket/filter + +Be aware that all the quirks from rsync exclusion/inclusion rules carry over +into bitpocket. If you decide that you need such advanced configuration, make +sure that you understand those rules very well, and consider double checking +them before syncing by running `bitpocket list`. + + +### Slow sync callbacks + +When syncing takes more than 10 seconds (SLOW\_SYNC\_TIME setting) bitpocket +can fire off user provided command in background. This can be usefull to notify +user about long sync happening, preventing him from turning off the machine +during sync etc. + +There are 3 settings that can be enabled in _.bitpocket/config_ file: + + # SLOW_SYNC_TIME=10 + # SLOW_SYNC_START_CMD="notify-send 'BitPocket sync in progress...'" + # SLOW_SYNC_STOP_CMD="notify-send 'BitPocket sync finished'" + +Just uncomment them and change at will. + +You can show tray icon during long sync with +[traytor](https://github.com/sickill/traytor) and following settings: + + SLOW_SYNC_START_CMD='~/bin/traytor -t "BitPocket syncing..." -c "xdg-open ." .bitpocket/icons & echo $! >.bitpocket/traytor.pid' + SLOW_SYNC_STOP_CMD='kill `cat .bitpocket/traytor.pid`' + + +## Author + +* Marcin Kulik | @sickill | https://github.com/sickill | http://ku1ik.com/ +* torfason | https://github.com/torfason -- cgit v1.2.3