Using Ansible with a bastion SSH host

A common network security design pattern is to prevent any connections to your application servers from outside of their private subnet, and then using a bastion host hosted in a DMZ to selectively whitelist traffic to the servers.

We have such a setup for one of our server pools through which we only allow SSH traffic from specific IP addresses. These servers are also provisioned via Ansible which programatically configures servers via SSH.

Due to our bastion host setup Ansible is unable to talk directly to our application servers so I had to find a way to proxy SSH connections through the bastion host.

I love using Ansible for creating simple tasks to run, for example flushing memcache to clear caches. Running with this example this is my Ansible structure:

devops/
    ansible/
        roles/
            memcache/
                tasks/
                    main.yml
                    restart.yml
        tasks/
            restart-memcache.yml
        vars/
            production-memcache.yml
    bin/
        restart-production-memcache.sh
    hosts.ini
    ssh.config
    ansible.cfg

The tasks/restart-production-memcache.sh script looks like this:

#!/bin/sh
ssh-add ${DEPLOY_KEYS_DIR}/memcache-servers.pem

ansible-playbook -i ansible/hosts.ini -u ansible ansible/tasks/restart-memcache.yml -v

It is called from the root devops folder and first adds the servers SSH key too the SSH agent and then calls the restart-memcache.yml playbook which in turn includes the memcache role restart.yml playbook (as well as other performing other tasks).

In the ssh.config file I have the following SSH configuration set:

Host bastion
    User                   ec2-user
    HostName               ###.###.###.###
    ProxyCommand           none
    IdentityFile           /path/to/ssh/key.pem
    BatchMode              yes
    PasswordAuthentication no

Host *
    ServerAliveInterval    60
    TCPKeepAlive           yes
    ProxyCommand           ssh -q -A ec2-user@###.###.###.### nc %h %p
    ControlMaster          auto
    ControlPath            ~/.ssh/mux-%r@%h:%p
    ControlPersist         8h
    User                   ansible
    IdentityFile           /path/to/ssh/key.pem

First the configuration for connecting to the bastion host is declared. Beneath that is a catch-all for all other hosts which in the ProxyCommand says to first connect to the bastion host then use netcat (nc) to pass the Ansible commands to the application server.

From the devops folder I can run ssh bastion -F ssh.config and I will be immediately connected to the bastion server.

Next Ansible needs to be told to use this custom SSH config when connecting to the application servers.

In the ansible.cfg file there is the following configuration:

[ssh_connection]
ssh_args = -o ControlPersist=15m -F ssh.config -q
scp_if_ssh = True
control_path = ~/.ssh/mux-%%r@%%h:%%p

When Ansible runs from the devops folder it will automatically pick up this ansible.cfg file and use the defined config when playbooks are run.

One issue with this setup is that Ansible's output as it runs is very verbose as it includes the SSH debug connection information as it passes through the bastion host to connect to the application servers; I've not yet found a way to supress this.

Changing Times

For the past year I've been relatively quiet about my day to day work owing to the "stealth" mode we have been operating with the project at Videogamer. After many a rewrite the initial MVP of the project is complete and I'm very proud of what we have designed, coded and architected. There is still some work to be done on the business side before the project is revealed but I'm really excited for the future of the site and the team.

What I can say is though is that I've had the opportunity to play with some great technologies such as Elasticsearch, Neo4J and Ansible and I truly feel excited to be a developer at the moment. In my spare time I'm playing around with Docker, and I've been teaching myself Ruby and RubyMotion and I've written about 5000 words of my book.

At the end of May my contract was up at Videogamer and it was time to move on, I've thoroughly enjoyed my time with the team and I wish them all the best for the future; I really do believe the platform we've been building is the foundation of something great and I'm excited to see it in the wild.

Tomorrow I'm joining Yuza, a mobile business incubator in central London as their new software engineer. Yuza have recently launched Pollen Velocity Capital, a service which pays developers their app store revenues on a weekly basis (instead of the 30+ days that it can take for Apple and Google to pay); this money can then be one-click reinvested into social media campaigns or paid directly into the developer's bank account.

I've been relatively quiet on the blogging front this year but I intend to get back to regular blogging soon - I've already a post about OpenID 2.0 in my drafts folder, and I've been working on a big update to my OAuth 2.0 Server library.

Until then, adieu.

Notification overload

Scenario:

Someone sends me a tweet, email, iMessage, or I get a calendar alert

Result:

  • My Macbook alerts me
  • My iPad alerts me
  • My iPhone alerts me
  • And now my wrist vibrates as the alert is forwarded from my iPhone to my Pebble.

Aaaghhh.

It would be great if Apple could solve the problem of sending notifications to whichever device is most relevant to me at the time - here are my suggestions:

  • If I'm sat casually browsing on my Macbook then alert me on there (but also send the notifications to my iPhone incase I suddenly get up and go).
  • If I'm working on my Macbook then send all notifications to my iPhone.
  • If I'm actively using my iPhone send all notifications there.
  • If I'm walking and my iPhone is sleeping in my pocket then buzz my Pebble.

A week with the Pebble

Just over a week ago my Pebble smartwatch arrived and I wrote up my initial thoughts. Having used it for a week now I thought I'd write up some further observations.

I said that my initial reason for buying the Pebble was so I could use it as actual watch and I'm slowly getting used to doing so though out of force of habit I still find myself reaching for my phone sometimes but I assume over time this will change.

One aspect that I hadn't thought about originally (but pointed out by Mother) was that of safety; wandering around late at night in areas of London I'm not as familiar with I do feel more comfortable quickly changing song with a few pushes on my wrist rather than pulling my phone out of my pocket and showing it off to anyone watching. I've already developed some muscle memory to quickly change song without looking at my wrist.

I've also developed some muscle memory when it comes to knowing where the buttons are and how much force to exert with my thumb in order to register a change of state in the menu. I still think you need to push a little too hard but I don't think a touch screen is necessarily a better solution.

I've now become used to having something on my wrist (this is the first watch I've owned) though I may change the strap at some point so I can have custom hole placements as I'm finding the two holes on the strap the Pebble comes with that I use the most are either ever so slightly too lose and so the watch slips or ever so slightly too tight so I'm conscious of that the watch is there.

The word Pebble is now part of my vernacular and everyone else around me because I keep yammering on about it. Therefore I've pretty much settled on not referring to it as a watch and just calling it "my Pebble".

I've been running beta versions of the 2.0 firmware and the better iOS app and there are some nice goodies that have been added:

  • Apps; I've got apps for getting bus, train and tube time. All immensely useful.
  • I've been trying the [7 minute workout exercise methodology] and there is an app for that.
  • I bought an app called Smartwatch Pro on the iOS AppStore which works as a "companion" to it's Pebble app. With this I can access my calendars, tweets, reminders, setup and execute custom HTTP requests (useful if you've got smart light bulbs for example) and also make my phone make a loud noise if I can't find it in my flat (I'm disgusted at myself for how much I've already used this - normally I have to eat into my Skype credit to call my mobile to find it).

Beta 6 of the firmware is still a bit buggy but I'm sure they'll iron out the kinks shortly.

All in all I'm still very happy with the Pebble, for the £90 or so it cost me it's provided excellent value for money and a definite place in my digital life.

Introducing OAuthello, a book about OAuth

https://s3.amazonaws.com/titlepages.leanpub.com/oauthello-a-book-about-oauth/large?1389476512

So tonight I announced on Twitter that I'm writing a book - OAuthello, a book about OAuth.

Why?

  1. I've been helping Phil Sturgeon edit his book for the past few weeks and I've really enjoyed the process.
  2. It's a personal challenge - I genuinely care about this subject and it motivates me.
  3. I've written several 5000+ word reports before in my old job so I know I'm capable of putting words on paper.
  4. Everyone else in the PHP community is writing a book so I may as well join in.

So here's my rough plan:

  1. Publicly talk about the book so people will badger me about it and judge me if I don't finish it.
  2. Get a few chapters done before I open the landing page up for sales.
  3. Finish the book by April/May.
  4. Talk about the whole process at Croydon Tech City once it's published.

Here goes!