Nicolas Ruflin

Measuring Your Software Quality Standards Must Be Easy

Quality in Software projects is important, as already elaborated upon in the previous post, it matters from day one. But as long as measuring and running quality metrics isn’t easy, it will be hard to follow through with quality.

For almost all programming languages there are tons of tools to run unit tests, check the code complexity or measure the code coverage. The easiest way to get these tools into your project is on day one. The longer you wait, the harder it gets to implement them. There will be dependencies which are not compatible, running the tests doesn’t work because you use a different folder structure and lots of other issues. The tools you choose to measure the quality will also influence the way you structure your project and your code. And I think this is good, as in most cases this leads to better structured code and less dependencies in the code which has lots of other advantages.

Setup and Build Tools

Installing the tools is not enough, it must also be easy to execute them and to get the metrics out. Otherwise, it is likely that you won’t use it and in case you share your project, others won’t use it. There are different tools out there such as Makefiles, Ant, Maven, Grunt and others, which allow you to automate running your tests or to get the code coverage. Pick the tool which is best for you and your project. For all tools, the execution should be just one command, such as make coverage in order to get the test coverage.

Not only running the tools must be automated, but also the setup of the tools. In case you have different dependencies in your project that are needed to build the test coverage, make sure that the setup of your project is also as simple as make setup. There are also lots of dependency managers out there that automate the installation of the dependencies for you, such as Bundler, Packagist or Pip. If you want to make it even nicer, make sure that your build script runs also make setup in case you run make coverage and some depenencies are missing …

To have these simple commands in place will make it very easy for you to run the quality metrics as often as possible. But also whenever new engineers join the project, with just one command they can start contributing to your project. If you don’t have these kind of scripts in place, you will spend a lot of time with every single engineer who joins the project. And in case some dependencies change, others don’t know how to get to the right setup again. This will not only cost you a lot of time, but it will also lead to frustration on all sides. To draw the analogy to building a car: Not having the script is like lending your car to someone else, but before the person is able to start driving, they either need a two-hour crash course from you or they need to read the full manual, because in your car, the accelerator is actually on the left side and the breaks are on the steering wheel. The next time the person will borrow your car, you changed the breaks to the left side and the whole process starts anew. If it is that hard to lend your car, you will not lend it to someone and the same is true for your code. If it is very hard for others to work with your code, you will not share it and they will not want to work with it.

Run it locally

It is crucial that all your quality measurements can be executed locally. How can you make sure your code is good enough if you can’t test it locally? When you have to commit your code to a build server first, which runs all the tasks and gives you feedback after several minutes / hours that you actually have one typo in a test, you will stop writing tests, since otherwise it will slow you down. In addition, I think “broken” code should never make it into a shared repository / branch, as otherwise you will not only break your own code, but everyone else’s, and your code will block the whole team. To draw the car analogy again: You implement a new break system in your car and also put it into the car the test team uses. After one week you realise that no test team is left, since all of them died in car accidents due to the breaks that didn’t work. You didn’t expect them to already use the breaks, since they were not finished / tested by yourself, but unfortunately they were testing the accelartor in the car which was ready for testing. Presumably your fellow engineers won’t die because of broken code, but frustration is guaranteed.

Make sure you and your team members can run all the metrics on your local machine and have it as close to the production system as possible. The best is to use a virtual environment with a clean setup. Tools such as Vagrant or Docker can make your life much easier. Do not run the code on your local machines, since normally every engineer has some special environment variables or tools installed on his computer. This can either get all the quality metrics to pass but then they fail on another system, or it can do the opposite. A simple example here is for all the engineers who develop on OS X. By default, OS X has a case insensitive file system. On the other hand, Linux, which most servers run on, does not. So in case you include the file image.png but it is actually called Image.png, it will probably work on your local machine but not on the production server. Run your code in a very simliar environment to the production in order to make sure it works. And again, make sure the setup of this virtual environment is automated so everyone in your team will use it.

Conclusion

In the optimal case, you would not even have to run your quality metrics. Instead, they would run every time you change your code and immediately give you direct feedback. There are tools such as watchr which are intended for continuous testing and which run the tests every time you change the code. Having feedback for your code as fast as possible is crucial in order to produce better code. Tests should not slow down your development but speed it up. Make sure that setting up your project and running your quality metrics is fully automated and that it happens as fast as possible. If your tests take a day to execute, no one will ever run the tests except your continuous integration system over night. This would mean the feedback cycle is one day long, which will make sure that you get a maximum of 365 feedbacks per year.

Links

Quality in Software Projects Matters From Day One

As I see quality in software as one of the most important features when building software, I will dedicate my next blog posts on software quality and quality standards. For most non-engineers, quality in software projects is something very abstract. I will therefore now and then make comparisons to building a car, since most people are more famliar with cars and objects they can actually touch and feel. My first post will be about why it matters to get quality into your software project from day one.

Quality in software projects is a topic that lots of engineers talk about and also embrace. I rarely witness discussions where engineers argue against the importance of quality. The only situation where the question of quality might be an issue is when talking about building a prototype or writing the first lines of code of a project. Should you have quality standards in your prototype? Should you start your project already with tests?

Software runs multiple times

My answer to this is: If you plan to run your software multiple times, you should. If you build a prototype for example during a hackathon that only needs to run once, perhaps you can build it without including some basic tests. But even in this situation, I belive that having some basic quality measures in your code will prevent nasty surprises during your presentation.

The beautiful thing about software is that it normally not only runs once, but can be executed thousands or millions of times without additional costs (except server costs). If you build software that is only executed once, why would you even build it? That is why quality standards also matter in your prototype. Your prototype will be executed lots of times and will go through various iterations. A prototype will have lower quality standards than a production system. To draw the comparison to building a car here: The first time you build a new car, you will not test 100 times if the blinker works as expected, but you will make sure that the breaks actually work. The same for your prototype, make sure you have at least tested the core functionality.

Proto-duction

More than once I have witnessed a prototype actually making it into production, the so-called Proto-duction. No, this should not happen, but if you start shipping your prototype and actual real users start to use it, it is very hard to convince the users and your business unit that you have to start from scratch again because you didn’t plan this software to actually be for users. If we make the comparison with the car again, we shippped it to the user and made sure the accelarator actually works as expected, as this is the first things users test, but we didn’t put much thought into how the breaks are applied if the car actually is going down and not on a flat street. Depending on how bad your “break” issue in your sofware is, perhaps it makes sense to take it away from your users again and build it from scratch.

Conclusion

To prevent Proto-Duction issues, get quality in your software projects from day one and make sure your quality is easily measured. More on how to make measuring your quality easy in the next post.

What HTTP Has to Offer Besides Code 418

The internet is an integral part of our daily lives. We would be lost without the internet at work, and we depend upon it to guide us to a good restaurant or hotel on our holidays. None of this would be possible without internet protocols. TCP/IP, DHCP, DNS, HTTP and many other internet protocols have been around for the last 20 years. Even more amazingly, they have changed only very little.

The largest part of the “visible” web has always been running on the Hypertext Transfer Protocol known as HTTP. HTTP/1.0 was introduced in May 1996 as RFC 1945. A few months later, HTTP/1.1 was released as RFC 2616. Since then, the protocol received some minor improvements, but stayed mainly the same.

In a fast-living environment such as the internet, 20 years of staying almost the same feels like several lifetimes. In their impact, protocols are comparable to inventing the wheel; both have been surving their purpose in many different environments and with different technologies. The inventors of HTTP, Tim Berners Lee and his colleagues, were very smart people who did not only think of what they needed back then, but also of how the protocol might be used in the future. If you study the RFC in detail, you see that HTTP does not only offer two, four or five request methods, but eight of them. Furthermore, it has a total of 41 status codes defined.

The downside of having something that has stayed the same for such a long time is the tendency we have to forget how good something actually is. As for the wheel, we need to take care we don’t reinvent internet protocols. Although the internet is all around us, only a small minority actually understands how it works. Most of them are engineers. And even engineers might have forgotten, or they might be younger than the protocols themselves and might never have learned it in the first place.

Nowadays, we are used to very quickly building small applications with API’s and frameworks. Seemingly, there is no need to understad the basic tools. Most engineers know at least the GET and POST Request methods and some of the status codes, such as 200 or 404. But most of us don’t have a deeper understanding, even though HTTP is at the core of most of the web applications we build.

So what about the status code 418? Code 418 is part of the Hyper Text Coffee Pot Control Protocol (HTCPCP/1.0) which was introduced as an April’s fool in 1998. Code 418 is therefore neither listed in the official RFC, nor should it be used as a status code. If it still haunts the internet, it is because we are able to use the tools to build applications, but we don’t always understand the tools themselves.

If we use a 418 code, or if we don’t know the difference between a 301 and a 302 code, we damage the internet little by little. Let’s re-learn HTTP and re-discover what it has to offer. For example, a status code that is not very well known but from my point of view is very usable is 202 (Accepted). Only by understanding internet protocols can we push them forward and be ready for HTTP/2.0, which is lurking on the horizon and will help us adapt to the new needs of internet communication.

Getting Started With the Internet of Things

The term Internet of Things has been around for quite some time. Until recently, however, the things were mostly offline and could only connect to the internet through a mobile phone, for example via scanning a QR code. This changed in the last months and years with quite a few start-ups that entered the market, such as Ninja Blocks or SmartThings and devices such as Knut or Twine, as well as larger companies such as Philips with the Hue Lightbulb. Some of them are based on standards such as Zigbee or have their own implementation.

The introduction of very cheap and simple computers such as the Raspberry PI and Arduino had a huge impact on the internet of things. The sensors that had until then been mostly offline can suddenly be connected to a cheap computer which connects to the internet. This makes it possible to access the sensors at any time from any location, mainly from the smart phone. It is even possible to interact with the sensor, for example turning the light on or opening a door.

Most of these sensors can only do one thing and are basically “stupid”. But the power is and will be added through software. As these sensors are now all connected to each other, it is possible to write applications that interact with the sensors and make the whole system intelligent.

From my point of view, the Internet of Things just got started. The more the things disappear and the more intelligent the interaction is, the more it will help us in our daily lives, and we will completely forget how it was possible to live without it. It will take a few years until these kind of things will be built into houses, but I think we are now at a good point to finally get the Internet of Things running.

After playing around with a Raspberry PI for quite some time, I finally ordered a Ninja Blocks Kit. I decided to go with the Ninja Blocks Kit for different reasons. First, I was only looking for a humidity sensor which I could monitor remotely. This is offered by various providers. What I like about the Ninja Blocks is that it is all based on Open Source. Not only the software is open source, but also the hardware plans are on Github. The product is at the moment probably more focused on Geeks than normal end users, but that’s how the whole thing starts. I would predict that in the next months there will be lots of small startups which build up on this basic infrastructure/service to provide end user friendly solutions for all kinds of stuff such as home monitoring, gardening and lots more.

I really look forward to getting my Ninja Blocks Kit. As soon as I get it I will post an update/review.

From Joomla to Octopress

After more than a year, I finally managed to upgrade my private website / blog. My blog and gallery used to be based on Joomla, as I was a contributor to Joomla some years ago and had built some extensions which were running my site. Every time Joomla released a large update I ran into trouble because either I was too lazy to upgrade my extensions or another external extension didn’t work properly with the upgrade. Naturally, it always took me forever to upgrade my website (even for security updates) and when I did upgrade, some stuff was broken.

Finally more than a year ago, I decided to switch to something simpler, perhaps even something that I didn’t have to host myself. I tried different services like Tubmlr, Posterous, Wordpress and more. These are all great solutions and make your life easier when you only want to blog. What bugs me with all these solutions is that it’s again very hard to move to a different service in case one of these services stops working — like Posteurous recently.

I prefer to write my blog post not in a WYSIWYG editor even if it supports raw text. From time to time, I want to insert some JavaScript or other things that the editors manage to break. So I was looking for a solution where I can write blog posts in a standardized format (HTML, Markdown) if possible in my preferred editor. Because of Github pages I stumbled over Jekyll. At first, I was very sceptical as I thought it is too limiting. On my old blog, I had some fancy extensions such as gallery and other stuff.

From Jekyll I moved to Octopress, as it offers some nice additions to Jekyll. What I really like about the solution is that it allows me to define my urls, have pages and posts and makes it really easy to deploy. The first plan was to migrate all blog entries from the old blog to the new one. As these were in two different languages and I couldn’t find an import script for Joomla, I decided to only migrate some blog entries related to Elastica and start from scratch.

So here is the new clean blog which will hopefully be filled with content again soon. At the moment I’m really happy with the new solution. Is is very easy to create blog entries and putting content online is just one command.

In case you miss some old blog entries you would like to have them online again, please let me now by sending a tweet to @ruflin.

Include Elastica in Your Project as Svn:externals

As most of you know, Elastica is hosted on github, which means it uses git as its revision control system. I have several projects which include Elastica but use subversion as its version control system. Until now, I included Elastica as an external svn source by hosting my own Elastica svn repository. But yesterday I discovered that the code from github can also be checked out through svn. I immediately asked google to get more details about this feature and discovered several blog entries on the github blog which I had somehow missed.

It is not only possible to check out repositories, but also to check out some specific subfolders or tags and you can even commit to the repository (which I didn’t test). As in my projects I only use the Elastica library folder and don’t need all the tests and additional data, I check out only the lib folder. If you want to check out the Elastica lib folder from version v0.18.6.0, use the following line of code:

1
svn co https://github.com/ruflin/Elastica/tags/v0.18.6.0/lib/ .

If you have a lib folder in your project with all your frameworks and libraries and you want to add Elastica as an external source (which is quite useful), you can set the svn:externals property on your library folder to the following.

1
https://github.com/ruflin/Elastica/tags/v0.18.6.0/lib/Elastica Elastica

If you already have other sources added as externals to your repository (for example ZF), just add this line below your existing lines. The next time you will update your repository, the Elastica folder with all its files will be checked out. To update to one of the next versions of Elastica, update the version number in the url in your svn:externals properties.

Using Elastica With Multiple Elasticsearch Nodes

Elasticsearch was built with the cloud / multiple distributed servers in mind. It is quite easy to start a elasticsearch cluster simply by starting multiple instances of elasticsearch on one server or on multiple servers. Every elasticsearch instance is called a node. To start multiple instances of elasticsearch on your local machine, just run the following command in the elasticsearch folder twice:

1
2
./bin/elasticsearch -f
./bin/elasticsearch -f

As you will see, the first node will be started on port 9200, the second instance on port 9201. Elasticsearch automatically discovers the other node and creates a cluster. Elastica can be used to retrieve all node and cluster information. In the following example first the cluster object is retrieved (Elastica_Cluster) from the client and then the cluster state is read out. Then all cluster nodes (Elastica_Node) are retrieved and the name of every node is printed out. Every cluster has at least one node and every node has a specific name.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$client = new Elastica_Client();

// Retrieve a Elastica_Cluster object
$cluster = $client->getCluster();

// Returns the cluster state
$state = $cluster->getState();

// Gets all cluster notes
$nodes = $cluster->getNodes();

foreach ($nodes as $node) {
    echo $node->getName();
}

Client to multiple servers

As elasticsearch is a distributed search engine that can be run on multiple servers, it is possible that some servers fail and still, the search works as expected as the data is stored redundantly (replicas). The number of shards and replicas can be chosen for every single index during creation. Of course, this can also be set with Elastica through the mapping as can be seen in the Elastica_Index test. More details on this perhaps in a later blog post.

One of the goals of the distributed search index is availability. If one server goes down, search results should still be served. But if the client connects to only the server that just went down, no results are returned anymore. Because of this, Elastica_Client supports multiple servers which are accessed in a round robin algorithm. This is the only and also most basic option at the moment. So if we start a node on port 9200 and port 9201 above, we pass the following arguments to Elastica_Client to access both servers.

1
2
3
4
5
6
$client = new Elastica_Client(array(
  'servers' => array(
      array('host' => 'localhost', 'port' => 9200)
      array('host' => 'localhost', 'port' => 9201)
  )
));

From now on, every request is sent to one of these servers in a round robin type. Instead of localhost, an external server could be used in addition. I’m aware that this is still a quite basic implementation. As probably some of you already realized, this is no safe failover method, as every second request still goes onto the server that is down. One idea here is to give a specific threshold for every server in which the respond time should be and otherwise the query goes to the next server. In addition, it would be useful to store this information on unavailable servers somewhere in order to use it for the next request. Thus, only one client has to wait for the unavailable server. Storing this information is somehow an issue, since Elastica does not have any storage backend.

Load Distribution

This client implementation also allows to distribute the load on multiple nodes. As far as I know, Elasticsearch already does this quite well on its own. But it helps if more than one node can answer http requests. Therefore, the method above is really useful if you use more than one elasticsearch node in a cluster to send your request to all servers.

It is planned to enhance this multiple server implementation in the future with additional parameters such as priority for a server and some other ideas. Please feel free to write down your ideas in the comment section or directly create a pull request on github.

How to Log Requests in Elastica

In the Elastica Release v0.18.4.1, the capability to log requests was added. There is a general Elastica_Log object that can later also be extended to log other things such as responses, exceptions and more. The Elastica_Log constructor takes an Elastica_Client as param. To enable logging, the config variable log for the client has to be set to true, or a specific path the log should be written to. This means that every client instance decides on its own whether logging is enabled or not.

The example below will log the message “hello world” to the general PHP log.

1
2
3
$client = new Elastica_Client(array('log' => true));
$log = new Elastica_Log($client);
$log->log('hello world');

If a file path is set as the log config param, the error log will write the “hello world” message to the /tmp/php.log file.

1
2
3
$client = new Elastica_Client(array('log' => '/tmp/php.log'));
$log = new Elastica_Log($client);
$log->log('hello world');

If logging is enabled, all request are at the moment automatically logged. There is a special conversion of request to log messages. The log message is converted to the shell format, so every log line can directly be pasted into the shell to test out. This is quite nice to debug and to create a gist if others ask what the query looks like. Furthermore, this makes it simpler to figure out whether the problem relates to Elastica or not.

For example the output for updating the number of replicas setting request for the index test would look like below.

1
curl -XPUT http://localhost:9200/test/_settings -d '{"index":{"number_of_replicas":0}}'

Storing and Analyzing Social Data

From July 2010 to December 2010 I worked on my master thesis “Storing and Analyzing Social Data”. It is about the structure of social data, how to store social data (i.e. NoSQL solutions) and the processing of social data. You can download the full thesis here.

For a short overview, here is the abstract of the thesis and the embedded pdf.

Abstract

Social platforms such as Facebook and Twitter have been growing exponentially in the last few years. As a result of this growth, the amount of social data increased enormously. The need for storing and analyzing social data became crucial. New storage solutions – also called NoSQL – were therefore created to fulfill this need. This thesis will analyze the structure of social data and give an overview of cur- rently used storage systems and their respective advantages and disadvantages for differently structured social data. Thus, the main goal of this thesis is to find out the structure of social data and to identify which types of storage systems are suit- able for storing and processing social data. Based on concrete implementations of the different storage systems it is analyzed which solutions fit which type of data and how the data can be processed and analyzed in the respective system. A focus lies on simple analyzing methods such as the degree centrality and simplified PageRank calculations.

Master Thesis

Storing and Analyzing Social Data