Some days ago we sent an internal quality report under confidentiality to our partners. After receiving a lot of requests to be allowed to publish this or parts of, we decided to publish the contents as it is. Yes, that’s right, the original confidential contents.
So what does report mean and how should you interpret this? Lets answer some questions:
For who was it written for?
Well, frankly, the report was actually really only for internal use. So that management and other departments in OXID could get an update on the quality status. Then somebody had the bright idea that this may interest the partners. We did that and .. the rest is history as they say .. now it’s public 🙂
Who wrote it?
The Quality Manager, Dainius Bigelis, responsible for the eShop quality assurance.
What does it mean ?
Basically this report presents some statistics which are important for quality management of the OXID eShop.
Ok, so what’s in it, then?
First, it’s about the source code in general. Here are some numbers about how large the code actual is, and how much of that is comments (more comments means better documented).
Why is source+comment <> total?.
Because empty lines are counted in the total but its not separately listed here.
Why is the count different between PE and CE? Aren’t they 100% the same ?
Main difference comes from license comments in each file. The CE file comment is larger then the PE.
Especially interesting is the dispersion over time as shown here.
Unit testing: Source code coverage means how much percentage of the code is covered with unit tests. Lets see what we have:
Currently coverage of eShop source code by unit tests is:
- EE 88.28%
- PE 90.59%
- CE 90.93%
and it’s continuously rising. BTW, we use PHPUnit for the PHP unit testing. You will have code which is difficult to unit test. We are at the 90% border. Ask the unit tests experts, they will confirm that this is very good! We have now about 3000 unit tests per edition.
As you can see the number of code in unit testing and selenium testing is growing consistently. Meaning better quality 🙂
Ok, the good is tested well. But is the code itself good maintainable and easy adaptable? You cannot tell that by looking at the size, or numbers of tests, can you?
No, that’s why we also check the cyclomatic code complexity and style convention errors of our code.
Cyclomatic complexity: This is a very interesting diagram! It gives a good indication if your code is to complex. How is it measured? Simple said, the more independent paths through a function, the more complex it is. See http://en.wikipedia.org/wiki/Cyclomatic_complexity for more details. We defined 20 as to much. Actual experts arguing, that with PHP 30 to 35 is not that bad either. We decided to go for 20 firstly. Our software should also be understandable by developers who just want to use the eShop and don’t want to spend much time to read it. As you can see nicely in the diagram, we found that over 60% of our old version of eShop broke this rule and also that we quickly refactored it to much lower levels. During development it raised a bit but in the quality assurance phase it lowered again. So what about the last couple of % ? Some functions do have good reasons to be higher.
Code conventions that important? Yes, they are, we want good readable code.
Here you can see how many bugs we had, how many are fixed, which severity and status dispersion they have. The diagram below shows the bug count cumulative over time. The red line is the most interesting one, it shows how many opened bugs we have at any time. I think this speaks for itself.
So, what’s the conclusion ?
I think we can safely say that the eShop code base has very good quality and have improved immensely compared to the previous version. Something to be proud of :-).