Thus far, I’ve not done anything serious with much more with database like Mysql, some Postgres and on the NoSql side MongoDB (with a frisson of some use of Redis for some barely-more-than basic things), but I saw some mention of using RedisGraph PHP Client back in late October, as part of my regular scan of the packagist feed for new PHP/Composer packages.

The ‘kjdev/redis-graph’ package is the first example of an interface library to RedisGraph – an extension module to Redis that became possible with Redis 4.0’s release. Other modules now include Bloom filters, rate-limiting and a JSON type.

This caught my interest. I’ve been a fan and user of Redis for several years, and it’s now a go-to tool that I use for most of my caching needs. I’ve not used it as a ‘serious’ database yet though, considering it more as a transient cache. That opinion is changing fast though.

I was curious about a few things:

  1. How fast to run a useful query?
  2. How fast is it to insert new (bulk) data?
  3. How much space does data take up?


Tagged with: ,

Slow running tests are a bug – they stop you from doing as much as you can with your code, & its quality. Spend a little time working on making your tests better, clearer, and faster – and you’ll reap rewards from your work.

I’ve had a couple of useful improvements in the time spent running my PHPunit tests recently.


In early 2016, I suggested an addition to the SncRedis-bundle. The project itself is an fully-featured add-on (‘bundle’) for Symfony framework projects to easily do a number of very useful interface functions between Symfony and the Redis database/cache.  It can, for example, quickly enable all the sessions to be put into a Redis server, and also cache the Doctrine meta-information as well as anything else that the developer would like to cache.

My additions were to do some relatively simple things – adding tags and aliases to the services it created from the configuration – in effect, really just adding some variables to the internal Symfony environment that could be used elsewhere, if required.

Now, it’s time to show you how why I thought it was useful to add, and what you can do with it.

You can do a lot with Redis, and when I’m using it, I prefer to keep things as separate as I can – internal (admin) caches, vs data that would be used on the public website, keeping the website sessions data apart from emails that are waiting to send, and background jobs that are being run.

With Redis, and the SncRedis bundle, it’s easy to keep them apart. While databases like MySql and Postgres give databases names, Redis prefers some simplicity – by default, it will create 16 numbered databases (0-15) – although telling the server to use more is just a quick config change away.

Recording which database is being used for which purpose though, it something that usually only happens in the configuration though – and so to show how much was in each numbered database, I kept a list to be able to refer to:

    0: Default
    1: Cache
    2: Admincache
    3: Swiftmail
    4: DoctrineMeta
    5: Sessions
    7: Queue


That’s easy enough to refer to or use, and also to keep up to date since it’s right next to the configuration. But, I’m a developer that loves to write code, and hates to keep documentation up to date, so I spent a few hours coding to allow the code to document the configuration.

That’s what the Symfony Aliases and tags allow for – it’s easy to fetch the data, and get the information about the database from Redis to be shown with what I’ve named database  – the ‘alias’ I’ve set in the configuration, and so now I can use that name to be able to find out more about the contents of the database. 

Here’s sample output of a fairly simple Symfony command I’ve written. If there’s no specific parameters of something to look at – it shows the list of databases and their aliases. Alternatively, I can use the name (here, ‘default’ or Redis dbNum:1) to get an item from the Redis database. Currently, that’s just an index into a dumped list – but it’s just as easy to get a named entry.

So, how can we get the details of the Redis clients, which refer to the databases inside Redis itself?  There are a few moving pieces:

  1. Add a CompilerPass – this sends the container, to some code that can collect data from it, before it is finished which will remove unused services and information.
  2. In the CompilerPass, find all the services tagged with the snc_redis.client‘ tag, and store the data required (in this instance, the Redis client, and the ‘alias’ that it is known as).
  3. Create a service to hold the data that the CompilerPass finds, and be able to retrieve it on demand

Lets go through them step class by class.

  1. Adding the CompilerPass – This is quite easy – we can create and add the relevant class right in the Symfony Framework’s core AppKernel.php class. You can also add the CompilerPass from within a bundle configuration, if you have any.
  1. The CompilerPass itself finds the tagged services (themselves created in SncRedis from the configuration), and arranges for them to be added to a holding service (the `AppBundle\SncRedis\ClientsList` class/service). Because each service could have more than one tag attached, we loop the list of potential tags, and have them added to the ClientList.
  1. Finally, the `AppBundle\SncRedis\ClientsList` class/service itself. A very simple store and retrieve. We can also get all the data, or just for a specifically named service, if it exists.

So – now we have a service that can store the data – how to use it? Get the service (which only then fills the information), get the clients and loop around to collect the parts of the information we need!

Once you have the name of the alias name of the queue, it’s simple to use that as part of the name of the snc_redis client service name, and so collect data about, or from the Redis Database service itself.

A recent post showed how to setup Code Tombstones – but there are other , even more insidious pieces of code in a project. The code you know you aren’t using now, but you wrote ahead of time – because  you think it will be useful, or you have plans for it, or any one of a dozen more reasons.

Chances are – you might never get back to it, and it’s just taking up important space. Disk space, or space in version control isn’t worth worrying about though – it’s the space in your brain that is most important for a developer.

You have to let it go.

Here’s the results from my own project – :

Before deleting unused code:

Classes: 30.22% (55/182)
Methods: 51.65% (515/997)
Lines: 40.99% (2080/5075)

After deleting code, and adding a few new tests

Classes: 34.52% (58/168)
Methods: 58.57% (523/893)
Lines: 52.18% (2299/4406)

… and all it took was deleting 2,615 lines of PHP classes, tests, Twig templates and yaml configuration – none of which were being used, or were in a state to use right now. The number of unit tests also dropped from 1,003 with 4,026 assertions, down to 984 and 3,966 assertions.

Because of the wonders of Git, and version control…

None of that code need ever be forgotten – I made a quick and simple branch – deleting-code branch and removed a block of related code (at least what I could find) per commit.

* 74a2e09 – (deleting-code) Removed unused commands
* 622295f – Remove deprecated code
* 4783f79 – Removing unused ContractorsList service
* 822e57b – Datatable & DemoTable routing and Javascript
* e28e535 – Remove RecruitersDemo – generated fake data on the fly
* 131864b – Removing the Mailable & Mailing code.
* e5b53b5 – Removing most of the ‘Fakes’ code
* ce8f717 – Start deleting unused code

I’ve also matched those commits with some issues, and plenty of searchable text that I can use to find them in the git log.

I, for example, I decide to come back and take another look at the ‘Mailable’ code (which is intended to make configuring and sending emails easier) – I can just search for ’emails’ or ‘mailable’ and revert just a single commit – and I’m right back where I need to be.  But for now, I want to just do it the ‘hard way’ with simple SwiftMailer-related code before I decide what can be done in an easier way.

While Twitter can be really annoying, sometimes it can help to promote some wonderfully simple ideas.

One of these came from Andrew Woods (@awoods) – a github repo called php-in-seattle.  It’s a simple idea – just a list of companies around a geographical area that use PHP.

What it can enable is of mutual advantage to the companies, and developers that might be looking for a new job. So, I started a similar list for the London (UK) area.

PHP-In-London, (UK)

There are a lot of companies in London that use PHP, and it is hard to keep track. If you know of any PHP companies with offices in the (Greater) London area (within ~20 miles), add them with a quick pull request (you can also edit directly from within the Github website if you want).

I’ve also added another list for recruiters that deal with a lot of PHP roles – though as they are somewhat less technically savvy, there are only a couple of entries, one of which is a recruitment startup.

Hacktoberfest T-shirtStarting the list actually helped to earn me a T-Shirt – for DigitalOcean Hacktober (which I’m actually wearing as I write this post :D).

To help spread the PR and Github Karma, when someone asked to add a new item to the list, I also added them as a full collaborator on the project – allowing them to add more – something that a some of them have already done – like these fine developers.


As a developer, I know well that anything worth doing, is worth testing, so I looked around and realised that there was indeed a way to test the URLs that were in the list. It’s called ‘Awesome_Bot’, and it is mostly aimed at the various ‘Awesome ‘ lists that helped to inspire the list in the first place. Setting it up with Travis isn’t hard, and so I’ve also enabled that.

It has taken a little tweaking of the configuration (even this morning, when I added Travis-CI to the whitelist to allow for redirects of the SVG build status button), but it has also helped to catch a couple of problems.

HacktoberFest 2017? Don’t wait till then, though!

Although HacktoberFest 2017 hasn’t been announced yet, if it does happen, I’m aiming to plug the project more in early October – but that doesn’t mean you have to wait until then. If you work at a company based in London (or at least easily commutable), take a look, and throw me a pull request! If you’ve not used Git/Github before, then let me know, and I can walk you through what to do, or worst case, just drop me an email with the relevant details and I’ll add them myself on your behalf.

One final question for you:

I’m considering a wider UK list of companies that use PHP, probably broken down by region and then larger towns/cities. If you think it’s a good idea – let me know with a comment, and/or Thumbs-Up (or down) on the issue I’ve created for the idea.

Should I create a separate PHP-in-UK repo?

Version 0.9 of scheb/tombstone autoloads a file with a tombstone() function. See the bottom of the post for a fix to override that in your own code.

In a large project – particularly one in a dynamic language like PHP, as a project gets bigger maintaining full control of the code can be difficult. New features are written, old ones are changed or deprecated. Sometimes code is left behind, unneeded in later versions, but still in the code-base.

This can be even more difficult in a full-stack framework like Symfony or Laravel, where some of the code is only run via the framework – such as Event Listeners/Subscribers, or the authentication layer.

Code test-coverage is good, but unit tests can’t tell you if the code *isn’t* being used elsewhere. You can spend time updating code, and the tests that just aren’t useful.

In late 2015, I came across an interesting potential solution – ‘Code Tombstones‘. They are a more refined version of an exception or simple echo/die. In production all they do is make a log entry – ‘I was here’. In a development environment, you might choose to immediately quit and complain – having the developer investigate if the code is genuinely useful or not..

Using a tombstone():

  1. Something that looks like a date – it’s just a string but it’s useful to note when you put the tombstone call in
  2. Your name, or initials – who put it there
  3. An (optional) label

They are all just simple strings that will just be output as-is if the function is called, but knowledge is power when it comes to tracking down what is happening.

You could also put the call into a branch (one choice of an `if` statement, for example). You can even put in into a PHP file outside of a class – though this will risk a large number of ‘ordinary failures’ because PHPunit (for example) will often read many files to find testing code, and so any code outside of a class in a file will be run, showing an error.

Defining the tombstone() function

This gets a little more interesting, because you may well want to have different configurations in different environments. In development or testing (including, for web-frameworks, like the Symfony app_test.php file) – a simple no-op – or you might want to send the messages produced to the console and then stop execution entirely.

In a production environment – the whole point is to just log and then continue.

In my prod-ready tombstone(), you see I also take the further step of catching any errors to then swallow them – you would not want testing code to break your live site!

There is one final thing I would suggest though – make sure you test that the tombstone will in fact write to the logfile on production. After a server move, I went several months not realising that any writes to the log was trying to write to a file that could not exist, but that error was being discarded. I put a quick failing-test URL that would force running the tombstone confirmed that the logfile was created as I had planned.

The library I use to write the logs comes from Christian Scheb, (scheb/tombstone), it takes care of the fiddly work of skipping enough of the debug_backtrace to figure out exactly where the function was called, and to log it, by as many log-handlers as you care to set.

Video from

David Schnepper’s Ignite talk, “Isn’t That Code Dead?” – Velocity Santa Clara 2014

Update: Dec 10th 2018 for version 0.9

As the scheb/tombstone library now autoloads a default tombstone() function, if you want to override that with a custom version (per environment for example), you’ll need to (potentially) move your versions in the web/app*.php files to above the require '../vendors/autoload.php'; line.

One of the advantages of a side-project is that you can be a little extra passionate about getting things just right. If you want to increase code coverage because you think that it’s good, you can – after all, it’s just some time now doing things that you like.

So, earlier in the year, when I saw Sebastian Bergmann’s article on ‘Questioning PHPUnit Best Practices’, I added it to a little (well, it currently stands at a count of 20 items…) list of clean-ups and improvements.

Well, a couple of days ago, I had the time to take a look at it, just a simple – almost relaxing – piece of work to convert from annotations-driven Exception testing to code-driven expectation tests.

The idea itself is quite simple – take a piece of code like such:

use App\CanThrowAnException;
use App\Exception\Complicated;

 * Pass the test if an exception is thrown
 * // assert what should be happening
 * @expectedException \App\Exception\Complicated
function testException()
    // setup the situation
    $systemUnderTest = new CanThrowAnException();

    // act

… to the newer style, ‘$this->expectException(…)’, which came in with PHPUnit 5.2.

use App\CanThrowAnException;
use App\Exception\Complicated;

 * Pass the test if an exception is thrown
function testException()
    // setup the situation
    $systemUnderTest = new CanThrowAnException();

    // assert what should be happening

    // act

This gives a couple of advantages –

  1. The test is contained entirely within the code. This means that the expectation of what should happen can be set after the initial setup has happened, giving a setup/assert/act direction of code
  2. Since it is now in PHP code, and no longer a annotation comment, it can use the namespace functionality and the ::class name resolution for shorter names, that will also still easily usable by automated PHP-parsing tools (For example, PHPStorm reads and understands the use lines to find all instances of the classes, even if they are aliased).

The rewriting from the annotation to function call is almost trivial, and there are also some new (related) function calls for the other @expectedException... annotations

@expectedExceptionCode -> $this->expectExceptionCode(int $code);
@expectedExceptionMessage -> $this->expectExceptionMessage($msgToExpect);
@expectedExceptionMessageRegExp -> $this->expectExceptionMessageRegExp($regexp);

The last thing I did before moving on was make sure that new @expectedException annotations didn’t creep back into the codebase. For that, I got a little ‘hack-y’. There are probably better ways to do it (with a PHPCS sniff), but I couldn’t find an easy way with a quick search – so I built my own.

First of all, a little history of how I develop software, and run tests. Back in 2013 I published what I called ‘personal-ci‘. It is, at it’s core an ant build.xmlcontrol file that runs a whole suite of tests. I’m still using the same system now, extending it very occasionally with new tools. I’ll run the full set of tests maybe a couple of times a day while a system is in frequent development and that will take about five to six minutes to run. A slightly smaller subset of the unit tests will only take two seconds though!

The difference in, in 5.5 minutes, it will also run a full php_codesniffer report, phpcpd, over 170 Behat integration scenarios, PHPUnit with code-coverage reports, PHPMetrics reports and Sami for documentation.

So, in my main build.xml file – I setup a couple of searches in the src/ and tests/ directories – if the string ‘@expectedException’ appears – fail. If one did stray back in, actually finding them is easy with ack or my own new favourite – ag.

<!-- If there are any '@expectedException...' annotations, fail. -->
<target name="noExpectedException">
    <property name="fail.string" value="@expectedException"></property>
    <fileset id="hasexpectedException_in_src" dir="src">
        <contains text="${fail.string}"></contains>
    <fileset id="hasexpectedException_in_tests" dir="tests">
        <contains text="${fail.string}"></contains>
    <fail status="1" message="One or more '@expectedException' detected">
                <resourcecount when="greater" count="0" refid="hasexpectedException_in_src"></resourcecount>
                <resourcecount when="greater" count="0" refid="hasexpectedException_in_tests"></resourcecount>

I’ve added that check to my php-linting action, which is run by default when ant is run.

<target name="php-lint-ci" depends="get-changeset.php.spacesep,noExpectedException" 
    description="Perform syntax check of sourcecode files in parallel">
    <!-- run `php -l` on the files for a quick safety check -->

So, that’s the latest addition to my ‘personal-ci’ workflow, updating to the latest best-practices in testing exceptions with PHPUnit, and making sure that the old-style annotations can’t be used in future.

Having just watched Sebastian Bergmann’s “The State of PHPUnit” presentation from Fosdem 2015, I was inspired to install and test a project of mine with the latest stable PHPUnit – v4.7. It was easily installed on the command line.

composer global require "phpunit/phpunit"

I installed it as a new, global, tool because in my project I am using the “ibuildings/qa-tools” repository to install and help run a number of QA tools – and the stable 1.1.* versions lock PHPunit to v3.7 – the last released version of which was in April 2014.

A good part of the reason to do so – beyond using the latest version – was also to enable the strict tests

    ... more parameters ...

This blogpost is to help someone else that tries it – and comes across the same issue I did:

PHP Fatal error: Class PHPUnit_Util_DeprecatedFeature_Logger contains 1 abstract method and must therefore be declared abstract or implement the remaining methods (PHPUnit_Framework_TestListener::addRiskyTest) in …vendor/phpunit/phpunit/PHPUnit/Util/DeprecatedFeature/Logger.php on line 201

My fix was simple – it took some systematic editing of the phpunit.xml file to figure it out. At first, I tried commenting out the various add-on I’ve got for PHPunit, tools to report slow tests, and to automatically close and check the results of any Mockery expectations. None of them helped, and so I started on the parameters in the opening XML tag of the file.

The actual fix was simple – the problematical line for me was:

colors = "false"

Removing that from the top of the phpunit.xml file, solved my issue, and now I’ve also gone on to update the “ibuildings/qa-tools” package to dev-master to get the latest-and-greatest (including automatically pulling in PHPUnit v4.* and Behat v3, among others). It was reassuring to know that I had the previous configuration safely stored in version control – so I could always just revert back to something that had worked. Running a separate copy of PHPunit installed outside of the project didn’t hurt either.

I’ve said for a long time that “you don’t get paid the big bucks for knowing what to do – it’s for knowing how to fix it when you make the inevitable screw-ups”.

Now, when I run my PHPunit-tests, I get a lot more warnings about ‘risky’ tests (all of it “This test executed code that is not listed as code to be covered or used”) – but those aren’t big issues for me right now.

The take-away is, don’t be afraid to upgrade, and if there is a problem, systematically (temporarily) commenting, or removing configuration, or code, can find the issues surprisingly quickly.

Tagged with: ,

Just a quick note to point out a couple of presentations on Queuing. I’ve recently shown the second (which admittedly has some significant things in common with the original, and not just the web-based slides).

Either way, you are welcome to look at them online, and the original html source, and some source code, are all online at

Tagged with:

It’s been one of those quiet spots around here for a while, so here’s the catch-up on what has been happening while I was not posting.

I’ve recently finished a short contract working with an agency, Transform (part of the Engine group) working with a couple of government departments. The Office Of The Public Guardian receives, checks and stores Lasting Powers of Attorney – a legal document that you write while still mentally compentant to say what you would like to happen should the worst occur, and by whom you want to do it. The simpler cases aren’t actually very complicated, but there is a lot of work to get the form completed – and the information can have to be written in triplicate over two or three different forms.

The project was to work with (and at the offices of) the new Government Digital Services (GDS) who are building the project, and I helped write the first-draft (but otherwise basically complete) prototype to put the form online. If nothing else, it allows someone to step through and only have to put information in once. Myself and one other developer, with a project manager and many others from the OPG and GDS took nearly all of the 37-pages of duplicated paper forms and created a PHP/Zend Form based system, that in the end produced PDFs, ready to check and then have signed by everyone involved.

It was an interesting project – and it will be a valuable service to make it easier to handle, and eventually also process from the back-office perspective. It’s not quite what I would normally do, I’m far more infrastructure and back-end oriented – not so used to building a large and complex flowing form, and so I elected to move on at the end of the prototype, rather than continue with the alpha/beta phases. With a little luck, it will go live later this year after some extensive user-testing to make it as useful, and easy as possible to fill in this important legal document. The end result, in a few years, should be the ability to have many times more people making an LPA for themselves, often as part as something as routine as making a will, or buying a new house.

Rest assured, there’s been plenty of relaxing since I finished that project a couple of weeks ago (especially since I spend a good portion of the time working while distinctly under the weather).

Puppet and Github

In the last couple of weeks, I’ve been a software developing machine. I’ve also been looking for my next contract role, hopefully something to start just after Easter – though, as I write this, exactly what that will be is still up in the air.

First, I’ve been putting together a development VM – currently based on a beta of Ubuntu 12.04. There are a few things that go into making it.

Puppet config
I’ve been occasionally tweaking this since before Christmas when I took some time to do a deeper dive into Puppet, after using it last year. Many of the modules I use, are actually pulled in from other github-based projects, especially from a number by (Steffen Zieger).

Puppet-dropbox: (forked from Dropbox is a very useful tool on any desktop to copy files around your own machines, and shared folders enable easy access to others working on the same project – I found it nearly invaluable in my last contract. It was also good to be able to improve the code that was upstream – a small fix to allow for Ubuntu to be also set as a destination for the installation.

In the end, I have elected to use it to just install the basic command line tool (rather than the full client), and then that can be used to install the main client, if required. It saves having to store the username and password in the repository, and it is also useful from other security standpoints not having copies of your files on all machines where the puppet manifests might be run.

Dotfiles is more of a meta-project, many appear to have a repo by that name, but a large number of them are hand-rolled. I forked one of the more common bases by Ryan Bates who also runs the excellent (which are not all about Ruby On Rails). I have yet to find a good way to integrate it with another shell-oriented project – Oh-My-Zsh, which is an excellent improvement over a standard Bash shell, that I’ve been using for more than a dozend years.

The common thread between both of these is to take a basic machine, with git, an SSH key and Puppet installed and bring it quickly up to a full spec development desktop/server. It’s a continuing project, but a valuable one, and not just as a learning tool.

There are two other projects that I’ve been working on.

guard-puppet-lint: Guard (see the railscast episode is a ruby-based project that will watch for file changes in a subdirectory hierarchy. There are a lot of plug-ins for it including PHP-oriented ones for PHPunit and PHP_CodeSniffer. The project itself can be downloaded from

As the name suggests, this small ruby gem adds a slightly easier way to run Puppet-Lint through Guard. As my first released ruby-code, there’s not much to it, and in fact, it’s really just a hack of guard-shell, that will run puppet-lint on the changed manifests. it does make it slightly cleaner though, and so I’m happy enough. I’m also very pleased to have had a couple of (very minor) issues raised – literally one word missing from the readme file and a single character to reduce the number of false-positive files that might be processed. There are some ideas I can add to it to make it even more useful, but that can wait for a little while, and besides, I have to figure out how to better use Guard in the first place, to be able to to do so.

My final, and latest, project is – a refactoring of the QR code library at

The code itself works fine, but only as a URL destination. One of the ideas that came up while working with the Office Of The Public Guardian on their new LPA form was to put QR codes onto the final output PDF pages to help verify automatically that all the pages that have been produced, have been recieved at the back office – and also being able to refer the paper form to a digital version stored in the database.

It’s a classic refactoring though – taking a piece of code and, without changing the end results, make it possible to use in a slightly different, but useful, context. Eventually, the qr.php webpage would be a thin wrapper around the class – and the class itself could be used from backend code to, for example, generate an image that can be placed into a PDF.

Tagged with: , , ,