Ways men in tech are unintentionally sexist

This post by Kat Hagan brilliantly summarises how men working in the tech industry are uninentionally sexist - http://notapattern.net/2014/10/14/ways-men-in-tech-are-unintentionally-sexist/.

On another point, last week I went to talk to the second and third year computer science students at the University of Lincoln about life as a software developer and some thoughts on the industry; it was great to see that there were significantly more women on the course than when I studied and worked there.

Introducing Proton

The last few big projects I've worked on (mostly backend RESTful APIs) have been built with a combination of different packages loaded with Composer instead of building on top of a framework such as Laravel or Symfony.

My go to packages to build the core of each project for a while now have been the following:

For each app I would spend time bootstraping the packages together and then build on that foundation. The bootstrap code isn't particularly excessive or laborious to put together, but it has been repetitive.

So as I started on another project this week I spend some time finally building a micro "framework" called Proton - https://github.com/alexbilbie/proton which binds and exposes the packages above with one reusable component

Here is a simple example of how to use Proton with anonymous functions:

And another with controllers:

Why did I build yet another micro framework? Why not just use Slim or Silex or some other micro framework?

Principally because I really like the particular packages above that I've already mentioned; I know their codebases very well and they're blazing fast. Also those other frameworks all offer features that I don't need; I like my code lean and mean and I don't want unnecessary overhead.

Proton contains just enough code to bootstrap those package components together (similarly to how I have done on previous projects), and leaves me to make the fundamental decisions about architecture myself; I'm not constrained in any way. Also the underlying packages are all exposed so I can work with the router, container and event dispatcher individually (as apposed to having to talk to them through a facade on the Proton\Application object).

Aside from this blog post I don't plan on publicising Proton particularly; I've written it to help myself out and if others find it useful then that's great too.

This week I also discovered StackPHP and it's completely changed the way I'm thinking about app development (and helped inform the design of Proton).

The principle of StackPHP is that you build up layers of classes that implement HttpKernelInterface. Then each implementing class decorates another class so that you get several layers that pass around Request and Response objects in a common way.

For example, before your application runs you might want to:

  1. Parse any cookies sent in the request
  2. Validate a session cookie that may be present
  3. Respond with a 403 if the session cookie is invalid and exit before the main application logic runs.

This is nicely demonstrated below by an onion image on the StackPHP website:

http://stackphp.com/img/onion.png

As long as each middleware (and the application itself) implements HttpKernelInterface you can literally stack these layers on top of each other and everything that follows will inherit and benefits from the preceding layers.

These middlewares can easily be with Proton too:

The StackPHP website has a number of useful middlewares built by the community, as well as some others that help with the decoration of the layers (StackPHP/Builder) and implement sessions in requests (StackPHP/Session). There are also four work-in-progress conventions and protocols that ensure interoperability between disparate middlewares.

I've already started on building a middleware league/oauth2-server and I will include this in the very nearly complete version 4 release.

Open Source Guilt

For the past few years I've "maintained" two very popular PHP libraries; an OAuth 2 Server and an OAuth 2 Client.

I've willingly and happily poured hours of my life into both projects. This was easily done when I worked for the University of Lincoln as both libraries were outputs of a project that I worked on. After leaving the university I moved to London and my life "got flipped-turned upside down" (as Will Smith once put it) which naturally resulted in a reduction in the number of commits that went into the projects.

This didn't concern me too much though; as far as I was concerned at the time the code worked, it was documented and I could point to a number of blog posts with solutions to common problems. I received numerous emails from other developers thanking me for my work on the projects and a number of developers in the PHP community that I respected promoted the library, invited me to talk on podcasts and offered me freelance opportunities to implement the libraries into their codebases. I was happy.

The second half of 2013 I reworked some of the server code which was quickly becoming the more popular of the two libraries and I released version 3 which improved the library's compliance with the OAuth 2.0 specification and the code itself was much cleaner. I tagged version 3, and I was again happy.

But I was lazy. I didn't update the documentation and I didn't update my blog posts which had example code implementations. Github issues started to appear asking for help, but some very kind developers who were interested in the project helped me answer them. The emails changed from thanks and congratulations to calls for help; and at one point I was receiving upwards of 30 emails a week.

Late 2013 I started working on version 4 to correct some pet-peeves I had with the code. I had become a benevolent dictator of the project and rejected numerous improvements to the existing codebase. I did my best with the emails piling up in my inbox but I also ignored many. My focus was only on the new version, because to me I knew it was better and I was able to easily implement it and I became frustrated by people who asked for help suggesting they just read the code and work it out.

During this time (from about November to March this year) I feel that on reflection I was fundamentally depressed and fed up. I felt terribly burnt out and couldn't even comprehend looking at a computer screen once I got home from work, I was unhappy with where I was living and I felt immense pressure because so many other developers were depending on me for help working with my code. Some people started to call me out publicly - and rightly so, others were rude and hurtful towards to me and I'm ever grateful to Phil Sturgeon for shooting them down for being arseholes. I'd hit a rut and I needed to get out of it.

Change started earlier this year when I moved house, and at the end of May I left my old job. I took a break between starting a new job, and I started exercising. I improved my diet, made some new friends and found enjoyment in coding again.

However I still had the issue of my projects. Others had worked on the client library and improved it, but my server library had got to about 35 open issues (most of which concerned the lack of documentation) and I'd not commented on them in weeks. I'd missed my promises to others of getting some examples out and actually documenting the current stable version. Another OAuth 2 server project also overtook mine in popularity (and rightly so).

About a fortnight ago I received an email from a developer which immensely upset me; he wasn't rude or angry but expressed his immense disappointment in me. I knew at that point that I had to get my act into gear and stop avoiding this elephant in the room.

And I've really tried. I've polished off version 4, merged it into the develop branch and finally written some goddamn documentation for it. As soon as that is finished (including an upgrade guide from version 3) I'm going to merge everything into master. Then I'm going to sort out the client library and bring that up to spec.

"With great power comes great responsibility"

Releasing open source projects is a great feeling however there are a number of considerations one should bear in mind:

  1. Actions have consequences - consciously making the decision to put something you've made on the web with an open source license with the expectation that others may find it useful has a side effect; other developers may come to you with questions, problems, and security concerns. You have a responsibility to do your best to help them out if you care about the integrity, security and reputation of your project.
  2. People want to help - others may come to you with suggestions to improve or change the codebase. Sometimes these changes can be great and you'll learn from them (I certainly have). Other times the change might be a good one but it may not fit in with your coding style, your future plans of the project or the code layout, and you should help the person making the change adapt it as necessary. Finally some suggestions may not be appropriate and you should politely explain why and offer suggestions as to how they can make more useful improvement.
  3. Your personal reputation is on the line - your name and email address will generally be embedded somewhere in the codebase, in git commit logs and in pretty graph visualisations on Github. This includes when you interact with others in bug trackers, and emailing others directly (nothing can stop them taking a screenshot and posting it on Twitter). If you act like an arsehole people will find it and call you out.
  4. Popular open source projects work well when the authors are using the project regularly themselves - one of the reasons why I have continued to work on the OAuth Server project more than the others is because I'm regularly implementing it in my personal projects or as part of my day to day job.

Based on my experiences in recent months I'm going to try the following to make my life easier and happier:

  1. I'm going to set out clear roadmaps for the project and more openly invite others to contribute to features.
  2. Document answers to the most common questions and problems so that I can point others to them in the future.
  3. Encourage people to ask questions - whilst this might seem counterproductive to my bulging inbox problem, I'm going to ask people to contact The PHP League's mailing list rather than contact me directly; this way multiple people can contribute answers.
  4. Be more honest about my availability to work on new features.

Deployment at Pollen

During my first few weeks at Yuza I've been working on one of our latest projects called Pollen which advances app developers' sales income every 7 days (rather than waiting up to 60+ days for Apple and Google to pay out).

The platform is hosted on AWS and I've spent some time perfecting our initial deployment process so that we can deploy safely and securely throughout the day.

This is our current setup:

  1. A developer works on a project locally, using provisioned Vagrant boxes which emulate the live environment.
  2. Code is merged into the develop branch using pull requests on Github.
  3. Our Jenkins box runs unit and behavioural tests against the develop branch when a change occurs.
  4. If everything is sound the code is merged into the master branch by Jenkins.
  5. Another Jenkins job watches for changes to the master branch. This job runs futher sanity checks, then pulls down dependencies, minifies assets and creates an archive which is pushed to an S3 bucket.
  6. Jenkins then tells AWS to spin up a new instance in a auto-scaling group (specific to the application type). The auto-scaling group uses a launch configuration which has the following "user-data" script which is executed when the instance spins up:The script installs Ansible and some other basic system utitilies, and automagically sets the hostname of the box using the AWS API. It also downloads pre-made SSH credentials which have specifc permissions to read certain Github repositories, one of which is our main devops repository. In this codebase there are shell scripts which will be executed depending on the hostname of the instance. These shell scripts will finalise the provisioning of the instance.
  7. When the instance is provisioned it downloads the latest code from the archive on S3, and is finally added to the load balancer. Jenkins will one by one terminate any other instances in the auto-scaling group which will in turn spin up new instances and eventually all instances in the group are running the latest codebase and setup.

These processes allows deploys to happen in a little under 5 minutes, and we'be designed all applications to be stateless so that they can be elastically scaled up and down as required.

For now this setup works well for now however there are some things we'd like to change.

Currently each new instance has to be provisioned from scratch which can be really slow if there is lots to install and setup; my plan is to get to grips with Packer. This will allow us to bake AWS AMI images seperately with all the appropriate software installed and configured. New instances will use these AMIs when they spin up, and so all that is required is to download the latest code archive from S3. Hopefully this should reduce the deploy time to no more than a minute or two.

I'm also going to keep my eye on Terraform which landed last week and is basically a way of describing a provider-agnositc cloud setup. This will then allow us to keep our code, provisioning scripts and cloud setup all in version control.

It's worth spending time examining and refining your deployment setup; it can significantly reduce onboarding time for new developers joining your team, and automating your deployment process improves productivity by reducing the number of repetative tasks required to ship your code.

Using Ansible with a bastion SSH host

A common network security design pattern is to prevent any connections to your application servers from outside of their private subnet, and then using a bastion host hosted in a DMZ to selectively whitelist traffic to the servers.

We have such a setup for one of our server pools through which we only allow SSH traffic from specific IP addresses. These servers are also provisioned via Ansible which programatically configures servers via SSH.

Due to our bastion host setup Ansible is unable to talk directly to our application servers so I had to find a way to proxy SSH connections through the bastion host.

I love using Ansible for creating simple tasks to run, for example flushing memcache to clear caches. Running with this example this is my Ansible structure:

devops/
    ansible/
        roles/
            memcache/
                tasks/
                    main.yml
                    restart.yml
        tasks/
            restart-memcache.yml
        vars/
            production-memcache.yml
    bin/
        restart-production-memcache.sh
    hosts.ini
    ssh.config
    ansible.cfg

The tasks/restart-production-memcache.sh script looks like this:

#!/bin/sh
ssh-add ${DEPLOY_KEYS_DIR}/memcache-servers.pem

ansible-playbook -i ansible/hosts.ini -u ansible ansible/tasks/restart-memcache.yml -v

It is called from the root devops folder and first adds the servers SSH key too the SSH agent and then calls the restart-memcache.yml playbook which in turn includes the memcache role restart.yml playbook (as well as other performing other tasks).

In the ssh.config file I have the following SSH configuration set:

Host bastion
    User                   ec2-user
    HostName               ###.###.###.###
    ProxyCommand           none
    IdentityFile           /path/to/ssh/key.pem
    BatchMode              yes
    PasswordAuthentication no

Host *
    ServerAliveInterval    60
    TCPKeepAlive           yes
    ProxyCommand           ssh -q -A ec2-user@###.###.###.### nc %h %p
    ControlMaster          auto
    ControlPath            ~/.ssh/mux-%r@%h:%p
    ControlPersist         8h
    User                   ansible
    IdentityFile           /path/to/ssh/key.pem

First the configuration for connecting to the bastion host is declared. Beneath that is a catch-all for all other hosts which in the ProxyCommand says to first connect to the bastion host then use netcat (nc) to pass the Ansible commands to the application server.

From the devops folder I can run ssh bastion -F ssh.config and I will be immediately connected to the bastion server.

Next Ansible needs to be told to use this custom SSH config when connecting to the application servers.

In the ansible.cfg file there is the following configuration:

[ssh_connection]
ssh_args = -o ControlPersist=15m -F ssh.config -q
scp_if_ssh = True
control_path = ~/.ssh/mux-%%r@%%h:%%p

When Ansible runs from the devops folder it will automatically pick up this ansible.cfg file and use the defined config when playbooks are run.

One issue with this setup is that Ansible's output as it runs is very verbose as it includes the SSH debug connection information as it passes through the bastion host to connect to the application servers; I've not yet found a way to supress this.