Author: mattions

Handpicked wordpress plugins for your self-hosted wordpress blog

A nice pic of a boat on a lake. Not really relevant with the post

A nice pic of a boat on a lake. Not really relevant with the post, but still pleasing

Intro

With the recent move from wordpress.com to a self hosted wordpress.org blog, I had the possibilities to pick some plugins that really have helped me to set-up and make the blog a tad bit more customized and close to my needs.

I’m gonna list them here, with a small description so it may be handy for someone else that is researching the subject as well

The handpicked Plugins

  • Worpress Importer This plugins lets you import your old wordpress.com blog in the new wordpress installation. Make sure you have `import everything` selected when you do the import, so all the images and attachments are happily downloaded and imported as well.You have to make sure you have increased the max upload file on you server, if you are importing a very big file. This is done changing you php configuration and the limit of max upload either in nginx or apache, depending what you are using. More info how to do this here.
  • Next plugin you want to get is JetPack from wordpress. This plugin has a lot of feautures that you can activate as you see fit. My favourite onese are: publicize (automatic sharing on G+, Facebook and Twitter), Moitor, that keeps an eye if your site goes offline and Photon, to serve images quicker from their CDN.
  • Spam is always a bad thing, and BruteProtect is a way to pretect yourself from it. You just activate it, and it is going to do is job.
  • Once you move to self-hosted blog, you have to manage also the backup for your site.
    A very handy plugin is UpdraftPlus – Backup/Restore, which gives you the ability to:

    1. Make automatic backup of your blog, including database, images, themes and plugins
    2. Upload you backup to a third party service, like for example DropBox
    3. Configure a schedule for your backup, with also a number of old backups you want to keep. My pick was 10 backups, with a weekly schedule.
    4. Restore your old backups with a single click.

    It’s very well designed and it works like a charm. Totally recommended

  • To make sure you write to the point, and keep your post interesting also for search engines, Worpress SEO is a good candidate. Although the title parser looks only for one keyword, so there will be always a disagreement between the plugin and a sane title, it’s extremely handy to keep sitemap up to date and automatically signal google when a new post pop up. Handy tool.
  • Due to the amount of code I tend to post, a nice way to present it, with proper highlighting it’s useful to have. For this I’ve picked Enlighter – Customizable Syntax Highlighter, which does a very good work, comes with themes to nicely integrate with the current palette of your site.
  • Last but not least, the Disqus Comment System is a nice and, according to me, superior way to enable comments on your post. It offers an import function to transfer all your old wordpress.com comments on the disqus system, and it’s pretty nice thing to have.

So there it is, some of the plugins I’m using on this website, which you may, or may not, find useful for your own site.

 

 

Dokku Environment variable special characters

Note to self: with dokku 0.3.1.7 special bash character in environment variable do not get parsed properly, and they break everything. Do not use them.

For example:

# this will fail
BLAH=!qwekcxzmpeqd('}

# this won't
BLAH=ewq989u0caad909ad

The amount of randomness involved is smaller. Most likely upgrading to 0.3.18 could fix this.

It took me 2 hours to figure this out and my push to my dokku server where denied.

Clean up old kernels

Do you want to save 15 Gb of space?

I have an old laptop from 2008, which is running ubuntu. Everytime a new kernel is released, this gets installed, however the older kernels and their image do remain available and they do not get automatically uninstalled. I guess this is a security feature, however if the installation is done only once, and years and years of new kernels are stacked, the space taken may start to be excessive and also big enough that reclaim it is a good idea.

Starting point

As you can see from below, my / partition was quite full

mattions@triton:~$ df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/sda1                 46G   43G  1.3G  98% /
udev                     2.0G  4.0K  2.0G   1% /dev
tmpfs                    396M  1.1M  395M   1% /run
none                     5.0M  8.0K  5.0M   1% /run/lock
none                     2.0G  260K  2.0G   1% /run/shm
cgroup                   2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/sda6                176G  166G  1.7G 100% /home
/home/mattions/.Private  176G  166G  1.7G 100% /home/mattions

The images stored in the boot partition were also taking quite a bit of space:

mattions@triton:~$ du /boot -sh
2.3G /boot

Purge the old kernels

To purge the old kernels, you can either do it by hand via ubuntu software center, or by synaptic, or use a script to do it for you. After a bit of googling, I’ve discovered the following script, aptly named purge-old-kernels, which I have also uploaded as a gist on git, just not to loose. Feel free to download it and use it if you want

When I ran the script, this is the list of the kernels that will be eliminated in my case (yours maybe will differ):


mattions@triton:~$ sudo bash Desktop/purge-old-kernels.sh Reading package lists... Done Building dependency tree Reading state information... Done The following packages were automatically installed and are no longer required: linux-headers-3.13.0-32 linux-headers-3.13.0-34 linux-headers-3.13.0-35 linux-headers-3.13.0-43 linux-headers-3.13.0-39 linux-headers-3.13.0-46 linux-headers-3.13.0-49 linux-headers-3.11.0-17 linux-headers-3.11.0-18 Use 'apt-get autoremove' to remove them. The following packages will be REMOVED linux-generic-lts-saucy* linux-headers-3.11.0-17-generic* linux-headers-3.11.0-18-generic* linux-headers-3.11.0-19-generic* linux-headers-3.11.0-20-generic* linux-headers-3.11.0-22-generic* linux-headers-3.11.0-23-generic* linux-headers-3.11.0-24-generic* linux-headers-3.11.0-26-generic* linux-headers-3.13.0-32-generic* linux-headers-3.13.0-33-generic* linux-headers-3.13.0-34-generic* linux-headers-3.13.0-35-generic* linux-headers-3.13.0-36-generic* linux-headers-3.13.0-37-generic* linux-headers-3.13.0-39-generic* linux-headers-3.13.0-43-generic* linux-headers-3.13.0-44-generic* linux-headers-3.13.0-45-generic* linux-headers-3.13.0-46-generic* linux-headers-3.13.0-48-generic* linux-headers-3.13.0-49-generic* linux-headers-3.13.0-51-generic* linux-headers-3.13.0-52-generic* linux-headers-3.2.0-23-generic* linux-headers-3.2.0-24-generic* linux-headers-3.2.0-25-generic* linux-headers-3.2.0-26-generic* linux-headers-3.2.0-27-generic* linux-headers-3.2.0-29-generic* linux-headers-3.2.0-30-generic* linux-headers-3.2.0-31-generic* linux-headers-3.2.0-32-generic* linux-headers-3.2.0-33-generic* linux-headers-3.2.0-34-generic* linux-headers-3.2.0-35-generic* linux-headers-3.2.0-36-generic* linux-headers-3.2.0-37-generic* linux-headers-3.2.0-38-generic* linux-headers-3.2.0-39-generic* linux-headers-3.2.0-40-generic* linux-headers-3.2.0-41-generic* linux-headers-3.2.0-43-generic* linux-headers-3.2.0-44-generic* linux-headers-3.2.0-45-generic* linux-headers-3.2.0-48-generic* linux-headers-3.2.0-49-generic* linux-headers-3.2.0-51-generic* linux-headers-3.2.0-52-generic* linux-headers-3.2.0-53-generic* linux-headers-3.2.0-54-generic* linux-headers-3.2.0-55-generic* linux-headers-3.2.0-56-generic* linux-headers-3.2.0-57-generic* linux-headers-3.2.0-58-generic* linux-headers-3.2.0-59-generic* linux-headers-3.2.0-60-generic* linux-headers-3.2.0-61-generic* linux-headers-3.2.0-63-generic* linux-headers-3.2.0-64-generic* linux-headers-3.2.0-65-generic* linux-headers-3.2.0-67-generic* linux-headers-3.2.0-68-generic* linux-headers-3.2.0-69-generic* linux-headers-3.2.0-70-generic* linux-headers-3.2.0-74-generic* linux-headers-3.2.0-75-generic* linux-headers-3.2.0-76-generic* linux-headers-3.2.0-77-generic* linux-headers-3.2.0-79-generic* linux-headers-3.2.0-80-generic* linux-headers-3.2.0-82-generic* linux-headers-3.2.0-83-generic* linux-headers-3.8.0-36-generic* linux-headers-generic-lts-saucy* linux-image-3.11.0-17-generic* linux-image-3.11.0-18-generic* linux-image-3.11.0-19-generic* linux-image-3.11.0-20-generic* linux-image-3.11.0-22-generic* linux-image-3.11.0-23-generic* linux-image-3.11.0-24-generic* linux-image-3.11.0-26-generic* linux-image-3.13.0-32-generic* linux-image-3.13.0-33-generic* linux-image-3.13.0-34-generic* linux-image-3.13.0-35-generic* linux-image-3.13.0-36-generic* linux-image-3.13.0-37-generic* linux-image-3.13.0-39-generic* linux-image-3.13.0-43-generic* linux-image-3.13.0-44-generic* linux-image-3.13.0-45-generic* linux-image-3.13.0-46-generic* linux-image-3.13.0-48-generic* linux-image-3.13.0-49-generic* linux-image-3.13.0-51-generic* linux-image-3.13.0-52-generic* linux-image-3.2.0-23-generic* linux-image-3.2.0-24-generic* linux-image-3.2.0-25-generic* linux-image-3.2.0-26-generic* linux-image-3.2.0-27-generic* linux-image-3.2.0-29-generic* linux-image-3.2.0-30-generic* linux-image-3.2.0-31-generic* linux-image-3.2.0-32-generic* linux-image-3.2.0-33-generic* linux-image-3.2.0-34-generic* linux-image-3.2.0-35-generic* linux-image-3.2.0-36-generic* linux-image-3.2.0-37-generic* linux-image-3.2.0-38-generic* linux-image-3.2.0-39-generic* linux-image-3.2.0-40-generic* linux-image-3.2.0-41-generic* linux-image-3.2.0-43-generic* linux-image-3.2.0-44-generic* linux-image-3.2.0-45-generic* linux-image-3.2.0-48-generic* linux-image-3.2.0-49-generic* linux-image-3.2.0-51-generic* linux-image-3.2.0-52-generic* linux-image-3.2.0-53-generic* linux-image-3.2.0-54-generic* linux-image-3.2.0-55-generic* linux-image-3.2.0-56-generic* linux-image-3.2.0-57-generic* linux-image-3.2.0-58-generic* linux-image-3.2.0-59-generic* linux-image-3.2.0-60-generic* linux-image-3.2.0-61-generic* linux-image-3.2.0-63-generic* linux-image-3.2.0-64-generic* linux-image-3.2.0-65-generic* linux-image-3.2.0-67-generic* linux-image-3.2.0-68-generic* linux-image-3.2.0-69-generic* linux-image-3.2.0-70-generic* linux-image-3.2.0-74-generic* linux-image-3.2.0-75-generic* linux-image-3.2.0-76-generic* linux-image-3.2.0-77-generic* linux-image-3.2.0-79-generic* linux-image-3.2.0-80-generic* linux-image-3.2.0-82-generic* linux-image-3.2.0-83-generic* linux-image-3.8.0-36-generic* linux-image-generic-lts-saucy* 0 to upgrade, 0 to newly install, 149 to remove and 1 not to upgrade. After this operation, 13.0 GB disk space will be freed. Do you want to continue [Y/n]?

Results

After I’ve choosen yes, it took quite a bit but at the end it was worth it

mattions@triton:~$ df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/sda1                 46G   28G   17G  63% /
udev                     2.0G  4.0K  2.0G   1% /dev
tmpfs                    396M  1.1M  395M   1% /run
none                     5.0M  8.0K  5.0M   1% /run/lock
none                     2.0G   37M  1.9G   2% /run/shm
cgroup                   2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/sda6                176G  166G  1.7G 100% /home
/home/mattions/.Private  176G  166G  1.7G 100% /home/mattions

And the Boot partition is just 70 Mb:

mattions@triton:~$ du /boot -sh
70M /boot

So from 45 Gb to 25 GB, and the boot partition itself from 2.3 Gb to 70 Mb.

Pretty good I think.

Google Chromecast review

google-chromecast

[easyazon_link identifier=”B00DR0PDNE” locale=”US”]Google Chromecast [/easyazon_link] is a nice hdmi dongle which you can connect to the tv and use it to stream all kinds of things from your phone, tablet and computer.

It is now from December that I have one, and I can say I am very pleased with it.

How does it work

The setup is pretty straightforward: you just need to connect to the tv, follows the screen instructions, and you are off to the races!

The dongle is controlled from your phone, and there are several apps that do interface with it directly, from example Netflix and Youtube.

The interesting bit is that your phone is controlling the dongle, like a remote control, however is not a screen share. Therefore you can use the phone at the same time you are streaming to the tv. This is a massive battery saver and gives you the ability to read the odd email if you need to, without interrupting the streaming.

Using google-chrome, and installing an extension for google chromecast, you have also the possibility to share any tab on the tv directly, which is very handy if you are looking at an embedded video.

Last but not least it is possible to share pictures and videos on the tv, a feature I used in a bunch of occasions.

TL;DR

I honestly think it’s a good value for money, and if you are watching youtube or netflix and you want an easy way to move it to the tv, the  [easyazon_link identifier=”B00DR0PDNE” locale=”US”]Google Chromecast [/easyazon_link] is what you want.

 

Hosting wordpress with dokku on digital ocean

Prologue

In this post I’ll describe how to host a wordpress blog on digital ocean and using dokku to deploy it.

As I have described in my last post my blog was hosted since few days ago on wordpress.com but there were two things that always annoyed me: the fact that I couldn’t use disqus for the comments and the annoying thing that wordpress was showing ads with me getting no money from it. My choices were either to pay money to get rid of them, or leave them be. (I was already paying for 13.00$ a year to have the blog at my custom domain.)

The other factor that made me change my mind about self-hosting, was that heroku is changing the free dyno in a not so free anymore dyno.

So there is only one decent response to this situation, explained in clear text by the image below:

 

let's dokku all the things

let’s dokku all the things

Dokku and co.

So you may think… what is dokku, what is heroku, what is a dyno, why on earth I ended up here?

Quick background information: Heroku is a platform that makes deployment very easy, and so far was the way to go solution to set up a small website. If your webproject is structured in a proper way (follows the buildpack spec) you can deploy with a simple git push.

This is extremely cool for people who do not want to install a webserver, or just activate some module on some obscure config file. Basically you can have the site up and running in no time. On top of this dokku uses docker as technology to run the instantiation of the website, which means it will be run in an isolated and lightweight way, without interfering with other apps running on the same host (a part of sharing resources). And you can scale up memory and CPU at least at will. So if you end up on hacker news or reddit, you could just bump it up and sustain more traffic. This is at least the theory.

Right now I’m working on a website, based on django which I’ve already deployed on heroku, but which I will transport on dokku, and I’ve taken the opportunity to host wordpress on my VPS as well, still using dokku. If you do not have a Virtual Private Server, I suggest to get one from Digital Ocean, ’cause they are pretty sweet.

How to get wordpress going on dokku

The important things you must know to host wordpress on dokku:

  • You need to create a database
  • You need to bump up the upload file size on your nginx configuration and php (to import your old blog, if you have one)
  • You need to make sure nginx accepts big post request to have JetPack working

To get Jetpack plugin fully working, you have also to tell php is running on port 80, otherwise he will get confused. You can do that editing your wp-config.php as shown below.

Ok so this are the steps, put together following several guides on the internet: mikai, ahmautom and shincoding. Note: at the time of writing the previous links are a bit outdated and I have made some modifications. So here it is how to host a wordpress blog on dokku updated to May 2015.

So these are the steps I’ve followed to get this blog going.

  1. Get the WordPress repo
  2. Setup the repo for heroku-php-buildpack
  3. Deploy check (no db connection)
  4. Add Database and volumes for plugins
  5. Follow the installation creating a new user
  6. Profit

Step 1: Get the WordPress repo

git clone https://github.com/WordPress/WordPress blog
cd blog
git checkout -b production # Create a custom production branch for our modification

Step 2: Setup WordPress to be served ad heroku-php application

You need to create a composer.json. At the moment mine looks like this:

{

"require": {
"php": "~5.6.0",
"ext-mbstring" : "*"
},

"scripts": {
"post-install-cmd": [
"chmod -R 777 wp-content"
]
}
}

I’m not sure about the scripts part, however it works, so I’ve left it in.

Create a Procfile with the following content:

web: vendor/bin/heroku-php-nginx -C nginx_app.conf --verbose

And add to the repo a nginx_app.conf file with:

# Custom nginx configuration #
##############################

# WordPress permalinks
location / {
index index.php index.html;
try_files $uri $uri/ /index.php?$args;

}

# Everything below here is optional, but recommended

# Add trailing slash to */wp-admin requests.
rewrite /wp-admin$ $scheme://$host$uri/ permanent;

# Deny access to any files with a .php extension in the uploads directory
# Works in sub-directory installs and also in multisite network
location ~* /(?:uploads|files)/.*.php$ {
deny all;
}

#upload
client_max_body_size 100M;

#jetpack connection
fastcgi_buffers 8 32k;
fastcgi_buffer_size 64k;
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;

# enable gzip compression
gzip on;
# Minimum file size in bytes (really small files aren’t worth compressing)
gzip_min_length 1000;
# Compression level, 1-9
gzip_comp_level 2;
gzip_buffers 4 32k;
gzip_types text/plain application/javascript text/xml text/css image/svg+xml;
# Insert `Vary: Accept-Encoding` header, as specified in HTTP1.1 protocol
gzip_vary on;
# end gzip configuration

# Set time to expire for headers on assets
location ~* .(js|css|png|jpg|jpeg|gif|ico|svg)$ {
expires 1y;
}

# Sitemap url, for WordPress SEO plugin
#rewrite ^/sitemap_index.xml$ /index.php?sitemap=1 last;
#rewrite ^/([^/]+?)-sitemap([0-9]+)?.xml$ /index.php?sitemap=$1&sitemap_n=$2 last;

Copy the wp-config.php.sample into wp-config.php and make sure you change all the variables into env-variables:

// ** MySQL settings - You can get this info from your web host ** //
/** The name of the database for WordPress */
define('DB_NAME', getenv('DB_NAME'));
/** MySQL database username */
define('DB_USER', getenv('DB_USER'));
/** MySQL database password */
define('DB_PASSWORD',getenv('DB_PASSWORD'));
/** MySQL hostname */
define('DB_HOST', getenv('DB_HOST') . ":" . getenv('DB_PORT'));
/** Database Charset to use in creating database tables. */
define('DB_CHARSET', 'utf8');
/** The Database Collate type. Don't change this if in doubt. */
define('DB_COLLATE', '');

/** Server settings for Jetpack */
$_SERVER['SERVER_PORT'] = 80;

/**#@+
* Authentication Unique Keys and Salts.
*
* Change these to different unique phrases!
* You can generate these using the {@link https://api.wordpress.org/secret-key/1.1/salt/ WordPress.org secret-key service}
* You can change these at any point in time to invalidate all existing cookies. This will force all users to have to log in again.
*
* @since 2.6.0
*/
define('AUTH_KEY', getenv('AUTH_KEY'));
define('SECURE_AUTH_KEY', getenv('SECURE_AUTH_KEY'));
define('LOGGED_IN_KEY', getenv('LOGGED_IN_KEY'));
define('NONCE_KEY', getenv('NONCE_KEY'));
define('AUTH_SALT', getenv('AUTH_SALT'));
define('SECURE_AUTH_SALT', getenv('SECURE_AUTH_SALT'));
define('LOGGED_IN_SALT', getenv('LOGGED_IN_SALT'));
define('NONCE_SALT', getenv('NONCE_SALT'));

Commit everything.

Let’s create the application on the server:

git remote add dokku dokku@:blog

and deploy it:

git dokku@ push production:master

This will create the application and do the first commit. If you have a domain pointing to the server where you have installed dokku, you can change you server ip with your domain ip.

If all is good you supposed to have an “Error establish connection to the database” when you try to reach the url where wordpress has been deployed.

Let’s set the salt for wordpress. Go to this URL and copy the values and set them as environment values (Do this from the server side, ’cause special character are not properly escaped when pushed on version 0.3.17 of dokku) similar to this:

dokku config:set blog AUTH_KEY='io2@kc=yepxhw+YN}Ufcl>|~Fes-`k-wzOH$JP23Wv{ivhzQj#9lco7VTGxnI6|r'
dokku config:set blog SECURE_AUTH_KEY=';=.]GhJdUCjZWo-&cu>9-rc{8Jxk7m0h+WXcK[K7BJ$0#/.@}k~E7AK!n;:b:9+4'
dokku config:set blog LOGGED_IN_KEY='}ZwMhvKCN-.]#(?^Vm>_,Lk!c90kZYzpEf7r+cw^v0nk6.mCV2eNVm]dw9.{5._u'
dokku config:set blog NONCE_KEY='Ei%OgQF)ecp:tIbW+eXx-[f^^JU,2 :)48eCPY/~rlMC=Wzt-j)ue`cJ}K&9b5k0'
dokku config:set blog AUTH_SALT=txKp#s~0xI#|cFfIE)9/A/q WkIQQR;$pv,Z/|eBo..~K_[{a@l~+{I5o6<~k!u'
dokku config:set blog SECURE_AUTH_SALT='V-|j%!_5,gF<2JoY(^&GriNM9400%[rzs
dokku config:set blog NONCE_SALT=':`)%I[j^&~Ysi5bja(B

We will set the database value in the same way, but first we need to create it.

Let’s go to the next step.

Install database and volumes

Docker runs on a temporal file system and therefore all the modification made will go away when the container will be shutdown, which means at the next deployment.

The plugin for the database is the following.:

To install them, login in on your server and than

cd /var/lib/dokku/plugins

# Maria db
git clone --recursive https://github.com/Kloadut/dokku-md-plugin mariadb
cd mariadb/dockerfiles
git checkout master
dokku plugins-install

You may have to run the dokku plugins-install with sudo privileges.

Create the database, give it the same name of your application, so it will automatically link.

$ ssh dokku@YOUR-SERVER mariadb:create blog

-----> MariaDB container created: mariadb/foo

Host: 172.16.0.104
User: 'root'
Password: 'RDSBYlUrOYMtndKb'
Database: 'db'
Public port: 49187

It’s necessary to have volumes connected, so the data in this volume will persist among different deployments. In particular we will make three compartments: one for the plugin we are going to install, one for the themes and one for the uploads.

Latest version of dokku (0.3.17) at the time of writing offers the ability to pass docker options, and therefore map an host directory to a container. I found out that it does not work from the client side, so you may need to run this from the server side.

dokku docker-options:add blog run "-v /opt/blog/wp-content/plugins:/app/wp-content/plugins"

dokku docker-options:add blog run "-v /opt/blog/wp-content/themes:/app/wp-content/themes"

dokku docker-options:add blog run "-v /opt/blog/wp-content/uploads:/app/wp-content/uploads"

Edit:

You have to do the same operation also for the deploy stage:


dokku docker-options:add blog deploy "-v /opt/blog/wp-content/plugins:/app/wp-content/plugins" dokku docker-options:add blog deploy "-v /opt/blog/wp-content/themes:/app/wp-content/themes" dokku docker-options:add blog deploy "-v /opt/blog/wp-content/uploads:/app/wp-content/uploads"

Thanks to yazinsai for pointing it out in the comments.

Ok, now we are pretty much done!

You new a new redeploy to make sure the volumes are picked up.

ssh dokku@YOUR-SERVER ps:rebuild blog

Another Edit

If you get stuck with thumbnails not working and problems with editing your theme locally,
have a look at the comment from Suisse, that maybe can point you in the right directions.

Profit

Go to the url where your blog is now hosted and follow the wordpress installation to have it up and running automatically, with permalinks and everything else working!

Plotting a new course

uncharted territory

uncharted territory

 

This blog exists from quite a while: the archive go all the way back to 2006, and it has been written in two languages, Italian and English, with the decision to stick to English from 15 July 2011.

Untill now it was hosted at wordpress.com, for a while under a normal wordpress domain, and lately under a custom domain.

It’s from a bunch of days that I have moved the blog to a new location, being hosted on a digital ocean droplet using dokku. In an upcoming post, I will write an how-to about how to do it. This was gave me the opportunity to make some modifications and start to update/change few things.

For example, the astute reader maybe has already noted that the blog is now sporting a fancy new theme and that all the comments are now powered by Disqus. The old comments have been imported and I hope this will suit people as well. Disqus is easier and better than the messy wordpress comments.

When it was running on wordpress.com, some google ads were shown automatically by worpress.com. To remove them, a user should have paid additional money. I never did it.

I think, now that I’m self-hosting, I may run some google ads and some amazon affiliate program myself, but I still did not make my mind up.

On one hand I’m interested to test them, to familiarize with some new technologies that I did not use so far, on the other hand I’m running an AdBlock software all the time, so I would be oblivious to this ads, as I guess the 80% of the readers that do show up here, which tend to be extremely technical and looking for answer.

More than Google ads, I think the interesting bit would be Amazon affiliate, given the fact I tend to do review of technology gadget or book quite a lot, but I do not really write about it. I guess time will tell.

As usual, expect all kind of topics, and maybe a bit more frequently.

Comments are open, as usual.

Bunch of useful django packages

Just a shout-out of some packages I’m using when creating a new django project

Following is the presentation I have given at the Cambridge django meetup, enjoy.

List intersection in python: let’s do it quickly

g8au5

So you have two lists and you want to make the intersection of it.
So you think about it for 3 seconds and than you write something like:

a = [1,2,3,4,"B"]
b = [2, "B"]
c = []
for e in a:
    if e in b:
        c.append(e)

This works, it seems very idiomatic and you’re done with it.
The problem this is extremely slow.

In other words writing a loop in python is a bad idea.
Why is slow? Because Loops in python are slow. Extremely slow.

If you have numbers only, I suggests to check out Numpy and even with string you can check pandas dataframe.

However if you have a mixture of object like above, you can just stick with python datastructure and use sets. If you do not have duplicates you’re out of luck…

With sets it will look like:

a = [1,2,3,4,"B"]
b = [2, "B"]
sa = set(a)
sb = set(b)
c = sa.intersection(sb)

For yours and my convenience, I’ve written a little gist to time it and plot it.

https://gist.github.com/mattions/22e3fd090b0390451420

Let’s see the results: (timings in seconds)

list_timing set_timing
elements
100 0.000370 0.000011
1000 0.008075 0.000082
10000 0.477722 0.001216
100000 49.045367 0.016954

figure_1

So with 10000 elements, with a list takes ~ 0.48 seconds, and with a set 0.0012 seconds, with a 100000 elements a list takes 49 seconds, and the set operation 0.017.

Two Take Home Messages:

  1. If you are writing a for loop, you’re doing it wrong
  2. If you have to intersect or unify list, transform them to sets and use the built-in function.

 

2014 in review

As usual WordPress offers the annual report with a bunch of stats and some copy written by mokeys (I actually believe it is the same stuff over and over, with the consequent possibility that the monkeys are no more that busy writing this stuff.

Anyway, given the time of the year, let me seize the opportunity to write few things about this year.

I was extremely busy, and I’ve managed to post only once. However the post was about Coinduit, and the genesis of it, which you could read here, if so you wish. It’s cool stuff, and it’s about bitcoins. Have a stroll if you feel inclined.

When I’ve started this blog, the main objective was to write useful posts so I could find them later. It turned out that some of them have been also useful for the incidental reader. As the matter of fact the top 5 posts are about getting something which is very niche right, it is nice to see that has been achieved. From how to sort out Pull Request, to getting the figures position right with LaTeX. Although there were some old posts like statistical distributions with ipython  and profile a python application, that could be a very quick read.

I’ll see if in 2015 will post more. I’ll guess we will discover it in a year.

In the meantime, Happy New Year!

Click here to see the complete report with the monkeys copy and some stats!