Random notes trying to install s3fs and backblaze support on Linux

B2 support for duplicity

Must be installed with pip for python 2

apt-get install python-pip
pip install b2

apt-get install duplicity
add-apt-repository ppa:duplicity-team/ppa
apt-get update
apt-get --only-upgrade install duplicity

duplicity /mount/point b2://<box's application ID>:<key>@<box id>

Worth noting that the documentation on backblaze themselves is wrong, you must use your bucket’s app id, not your master app id

Installing s3fs on ubuntu

libfuse must be installed from source or else you’ll get “modprobe: ERROR: ../libkmod/libkmod.c:514 lookup_builtin_file() could not open builtin file ‘/lib/modules/2.6.32-042stab131.1/modules.builtin.bin'”

wget https://github.com/libfuse/libfuse/archive/fuse-2.9.8.tar.gz
tar xvf fuse-2.9.8.tar.gz
cd libfuse-fuse-2.9.8
./configure

Opps… error! 

zsh: no such file or directory: ./configure

I noticed there’s configure.ac file, so it should be

autoconf

Opps… error again

configure.ac:6: error: possibly undefined macro: AM_INIT_AUTOMAKE
If this token and others are legitimate, please use m4_pattern_allow.
See the Autoconf documentation.
configure.ac:10: error: possibly undefined macro: AC_PROG_LIBTOOL
configure.ac:13: error: possibly undefined macro: AM_PROG_CC_C_O
configure.ac:73: error: possibly undefined macro: AM_ICONV
configure.ac:75: error: possibly undefined macro: AM_CONDITIONAL

Let’s try this

> ./makeconf.sh
Running libtoolize…
libtoolize: putting auxiliary files in '.'.
libtoolize: copying file './ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
Running autoreconf…
configure.ac:10: installing './compile'
configure.ac:5: installing './config.guess'
configure.ac:5: installing './config.sub'
configure.ac:6: installing './install-sh'
configure.ac:6: installing './missing'
example/Makefile.am: installing './depcomp'

Cool, seems to work well

./configure
make
make install

Okay… now let’s try

>s3fs bucket /mount/point
fuse: failed to open /dev/fuse: Operation not permitted

Damn…

I’m going to use aws-cli instead. From https://gist.github.com/keeross/b19b6e5603c78073d58b

aws-cli to backup vesta to Amazon S3

  • Install pip (if not installed already):
$ curl -O https://bootstrap.pypa.io/get-pip.py
  • Install AWS cli tool:
$ sudo pip install awscli
  • Configure AWS cli tool:
$ aws configure
AWS Access Key ID [None]: <YOUR_AWESOME_KEY>
AWS Secret Access Key [None]: <YOUR_AWESOME_KEY>
Default region name [None]: us-west-2
Default output format [None]: json
  • Backup old file, then open and edit Vesta command v-backup-user:
$ cp /usr/local/vesta/bin/v-backup-user /usr/local/vesta/bin/v-backup-user_backup
$ sudo nano /usr/local/vesta/bin/v-backup-user
  • Somewhere in Variable&Function section add:
# AWS arguments
bucket_name=<your_awesome_bucket_name>
  • Search for # Creating final tarball section and right after this line chown admin:$user $BACKUP/$user.$date.tar add new code:
# AWS UPLOAD
aws s3api put-object --bucket $bucket_name --key $user/$user.$date.tar --body $BACKUP/$user.$date.tar --storage-class REDUCED_REDUNDANCY
echo -e "$(date "+%F %T") AWS: Uploaded and backed. $user.$date.tar"
msg="$msg\n$(date "+%F %T") AWS: Uploaded and backed. $user.$date.tar"

Error mapping file into docker container

Specifically, for Docker on Windows, Virtualbox version, in conjunction with WSL. When I run ‘docker-compose up’, I get this error

ERROR: for 125f20ccead0_lempr_web_1 Cannot start service web: OCI runtime create failed: container_linux.go:348: starting container process caused “process_linux.go:402: container init caused \”rootfs_linux.go:58: mounting \\\”/d/Code/lempr/web/nginx.conf\\\” to rootfs \\\”/mnt/sda1/var/lib/docker/aufs/mnt/065f6aacb3ee01072637ae0544f50a9b772df662ea8e7e3dc2d663d87d7e1a4f\\\” at \\\”/mnt/sda1/var/lib/docker/aufs/mnt/065f6aacb3ee01072637ae0544f50a9b772df662ea8e7e3dc2d663d87d7e1a4f/etc/nginx/nginx.conf\\\” caused \\\”not a directory\\\”\””: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

I checked to make sure nginx.conf exist both outside and inside the container.

Then I tried to map the file to a different location, it gets created as a directory inside the container!

I tried various suggestions from the internet:

  • Restart the host machine
  • Map /mnt/c to /c
  • Share the /c drive to the docker-machine VM inside virtual box
    • To do so, open virtualbox
    • Select the ‘default’ machine
    • Go to settings
    • Go to ‘shared folders’
    • Click add
    • Add C and D as shares
    • Attach a console to the machine to make sure the share works

To no avail. Meanwhile the same docker-compose works just fine on my Mac.

So I concluded WSL mapping and docker for Windows’ mapping can’t handle this use case at all and I should switch to a *nix OS for the task.

Windows seems to be unable to handle this setup

Docker-Machine (in Virtualbox) -> Windows -> WSL

It seems there’s a problem with mapping paths from the docker-compose VM to Windows, not a problem with WSL, since trying within Docker’s MINGW command line yields the same result as WSL’s.

 

How to install zip extension for PHP7

Specifically for LAMP stack / webmin

First, install the module

sudo apt-get install php7.0-zip

Then restart the web server

sudo service apache2 restart
sudo service nginx restart

That’s it!

Javascript idioms

From Zoltan Kochan, author of pnpm

 

Double exclamation

Prefixing anything with !! converts it to a boolean.

var foo = 0
console.log(!!foo)
//> false

Essentially it is a shorter way to write Boolean(foo).

Converting arguments to array

The arguments object can be used to access the arguments passed to the function. However, it is not an Array so it doesn’t have Array properties except length. The Array.prototype.slice.call(arguments) idiom is used very frequently to convert the argument’s object to an actual array.

(function() {
  console.log(arguments instanceof Array)
  //> false

  var args = Array.prototype.slice.call(arguments)
  console.log(args instanceof Array)
  //> true
})()

Assigning default values

function foo(opts) {
  var msg = opts.message || 'Hello world!'
  console.log(msg)
}

// instead of
function foo(opts) {
  var msg = opts.message ? opts.message : 'Hello world!'
  console.log(msg)
}

More examples of interesting || and && usages can be found in the 12 Simple (Yet Powerful) JavaScript Tips article.

Converting to array if not already

var totallyArray = [].concat(value)

//instead of
var totallyArray = value instanceof Array ? value : [value]

Converting strings to number

var foo = +'12.2'
var bar = +'12'

// instead of
var foo = parseFloat('12.2')
var bar = parseInt('12')

Checking if an array includes an element

if (~[1, 2, 3].indexOf(2)) { console.log('includes') }

// instead of
if ([1, 2, 3].indexOf(2) > -1) { console.log('includes') }

There are some other usage examples for the tilde operator as well in The Great Mystery of the Tilde(~).

Writing multi-line strings

var multiStr = [
  "This is the first line",
  "This is the second line",
  "This is more..."
].join("\n");

// instead of
var multiStr = "This is the first line\n" +
  "This is the second line\n" +
  "This is more...";

Looping through an array

It can be used if order is not important

for (var i = arr.length; i--;) {
  // ...
}

// instead of
for (var i = 0; i < arr.length; i++) {
  // ...
}

setTimeout(func, 0)

JavaScript code runs on one thread. Calling setTimeout with 0 allows to schedule a function to run after the current event loop tick.

setTimeout(function() {
  console.log('log message from next tick')
}, 0)

console.log('Hello world!')
//> Hello world!
//> log message from next tick

RabbitMQ REST API

There’s a little known REST API built into RabbitMQ, accessible via the management plugin. Its methods are as follows:

GET PUT DELETE POST Path Description
X /api/overview Various random bits of information that describe the whole system.
X X /api/cluster-name Name identifying this RabbitMQ cluster.
X /api/nodes A list of nodes in the RabbitMQ cluster.
X /api/nodes/name An individual node in the RabbitMQ cluster. Add “?memory=true” to get memory statistics, and “?binary=true” to get a breakdown of binary memory use (may be expensive if there are many small binaries in the system).
X /api/extensions A list of extensions to the management plugin.
X X /api/definitions
/api/all-configuration (deprecated)
The server definitions – exchanges, queues, bindings, users, virtual hosts, permissions and parameters. Everything apart from messages. POST to upload an existing set of definitions. Note that:
  • The definitions are merged. Anything already existing on the server but not in the uploaded definitions is untouched.
  • Conflicting definitions on immutable objects (exchanges, queues and bindings) will cause an error.
  • Conflicting definitions on mutable objects will cause the object in the server to be overwritten with the object from the definitions.
  • In the event of an error you will be left with a part-applied set of definitions.

For convenience you may upload a file from a browser to this URI (i.e. you can use multipart/form-data as well as application/json) in which case the definitions should be uploaded as a form field named “file”.

X /api/connections A list of all open connections.
X X /api/connections/name An individual connection. DELETEing it will close the connection. Optionally set the “X-Reason” header when DELETEing to provide a reason.
X /api/connections/name/channels List of all channels for a given connection.
X /api/channels A list of all open channels.
X /api/channels/channel Details about an individual channel.
X /api/consumers A list of all consumers.
X /api/consumers/vhost A list of all consumers in a given virtual host.
X /api/exchanges A list of all exchanges.
X /api/exchanges/vhost A list of all exchanges in a given virtual host.
X X X /api/exchanges/vhost/name An individual exchange. To PUT an exchange, you will need a body looking something like this:
{"type":"direct","auto_delete":false,"durable":true,"internal":false,"arguments":{}}

The type key is mandatory; other keys are optional.When DELETEing an exchange you can add the query string parameter if-unused=true. This prevents the delete from succeeding if the exchange is bound to a queue or as a source to another exchange.

X /api/exchanges/vhost/name/bindings/source A list of all bindings in which a given exchange is the source.
X /api/exchanges/vhost/name/bindings/destination A list of all bindings in which a given exchange is the destination.
X /api/exchanges/vhost/name/publish Publish a message to a given exchange. You will need a body looking something like:
{"properties":{},"routing_key":"my key","payload":"my body","payload_encoding":"string"}

All keys are mandatory. The payload_encoding key should be either “string” (in which case the payload will be taken to be the UTF-8 encoding of the payload field) or “base64” (in which case the payload field is taken to be base64 encoded).
If the message is published successfully, the response will look like:

{"routed": true}

routed will be true if the message was sent to at least one queue.Please note that the HTTP API is not ideal for high performance publishing; the need to create a new TCP connection for each message published can limit message throughput compared to AMQP or other protocols using long-lived connections.

X /api/queues A list of all queues.
X /api/queues/vhost A list of all queues in a given virtual host.
X X X /api/queues/vhost/name An individual queue. To PUT a queue, you will need a body looking something like this:
{"auto_delete":false,"durable":true,"arguments":{},"node":"rabbit@smacmullen"}

All keys are optional.When DELETEing a queue you can add the query string parameters if-empty=true and / or if-unused=true. These prevent the delete from succeeding if the queue contains messages, or has consumers, respectively.

X /api/queues/vhost/name/bindings A list of all bindings on a given queue.
X /api/queues/vhost/name/contents Contents of a queue. DELETE to purge. Note you can’t GET this.
X /api/queues/vhost/name/actions Actions that can be taken on a queue. POST a body like:
{"action":"sync"}

Currently the actions which are supported are sync and cancel_sync.

X /api/queues/vhost/name/get Get messages from a queue. (This is not an HTTP GET as it will alter the state of the queue.) You should post a body looking like:
{"count":5,"requeue":true,"encoding":"auto","truncate":50000}
  • count controls the maximum number of messages to get. You may get fewer messages than this if the queue cannot immediately provide them.
  • requeue determines whether the messages will be removed from the queue. If requeue is true they will be requeued – but their redelivered flag will be set.
  • encoding must be either “auto” (in which case the payload will be returned as a string if it is valid UTF-8, and base64 encoded otherwise), or “base64” (in which case the payload will always be base64 encoded).
  • If truncate is present it will truncate the message payload if it is larger than the size given (in bytes).

truncate is optional; all other keys are mandatory.

Please note that the get path in the HTTP API is intended for diagnostics etc – it does not implement reliable delivery and so should be treated as a sysadmin’s tool rather than a general API for messaging.

X /api/bindings A list of all bindings.
X /api/bindings/vhost A list of all bindings in a given virtual host.
X X /api/bindings/vhost/e/exchange/q/queue A list of all bindings between an exchange and a queue. Remember, an exchange and a queue can be bound together many times! To create a new binding, POST to this URI. You will need a body looking something like this:
{"routing_key":"my_routing_key","arguments":{}}

All keys are optional. The response will contain a Location header telling you the URI of your new binding.

X X /api/bindings/vhost/e/exchange/q/queue/props An individual binding between an exchange and a queue. The props part of the URI is a “name” for the binding composed of its routing key and a hash of its arguments.
X X /api/bindings/vhost/e/source/e/destination A list of all bindings between two exchanges. Similar to the list of all bindings between an exchange and a queue, above.
X X /api/bindings/vhost/e/source/e/destination/props An individual binding between two exchanges. Similar to the individual binding between an exchange and a queue, above.
X /api/vhosts A list of all vhosts.
X X X /api/vhosts/name An individual virtual host. As a virtual host usually only has a name, you do not need an HTTP body when PUTing one of these. To enable / disable tracing, provide a body looking like:
{"tracing":true}
X /api/vhosts/name/permissions A list of all permissions for a given virtual host.
X /api/users A list of all users.
X X X /api/users/name An individual user. To PUT a user, you will need a body looking something like this:
{"password":"secret","tags":"administrator"}

or:

{"password_hash":"2lmoth8l4H0DViLaK9Fxi6l9ds8=", "tags":"administrator"}

The tags key is mandatory. Either password or password_hash must be set. Setting password_hash to “” will ensure the user cannot use a password to log in. tags is a comma-separated list of tags for the user. Currently recognised tags are “administrator”, “monitoring” and “management”.

X /api/users/user/permissions A list of all permissions for a given user.
X /api/whoami Details of the currently authenticated user.
X /api/permissions A list of all permissions for all users.
X X X /api/permissions/vhost/user An individual permission of a user and virtual host. To PUT a permission, you will need a body looking something like this:
{"configure":".*","write":".*","read":".*"}

All keys are mandatory.

X /api/parameters A list of all parameters.
X /api/parameters/component A list of all parameters for a given component.
X /api/parameters/component/vhost A list of all parameters for a given component and virtual host.
X X X /api/parameters/component/vhost/name An individual parameter. To PUT a parameter, you will need a body looking something like this:
{"vhost": "/","component":"federation","name":"local_username","value":"guest"}
X /api/policies A list of all policies.
X /api/policies/vhost A list of all policies in a given virtual host.
X X X /api/policies/vhost/name An individual policy. To PUT a policy, you will need a body looking something like this:
{"pattern":"^amq.", "definition": {"federation-upstream-set":"all"}, "priority":0, "apply-to": "all"}

pattern and definition are mandatory, priority and apply-to are optional.

X /api/aliveness-test/vhost Declares a test queue, then publishes and consumes a message. Intended for use by monitoring tools. If everything is working correctly, will return HTTP status 200 with body:
{"status":"ok"}

Note: the test queue will not be deleted (to to prevent queue churn if this is repeatedly pinged).

This can probably be used to build a library that monitor RabbitMQ nodes and automatically select the best node to connect to