Quality Report for OXID eShop 4.6 has been published

We are happy to announce that Quality Report for OXID eShop 4.6 has been prepared and finalized by quality assurance team and is now published. (for download see below)

In our quality report we analyze and interprete the most important quality control measures of OXID eShop and draw comparisons between current and previous versions of the platform. OXID Quality assurance team makes big efforts to ensure that OXID eShop in every new version provides our customers with ever higher quality.

The results are reflected in the quality report: you will find some important software quality measures of OXID eShop 4.6.0. like code coverage, bug tracking /fixing process, bug status statistics, statistics about implemented improvements for performance, and also results of performance tests.

We are especially proud to announce that – despite the steady growth of the source code – we are able to keep the code coverage of our unit tests above 90%!

From now on we will continue to issue a quality report for each major and minor release. Reports appear some time after the release of the software so we can collect and evaluate more data from the final version. This way we hold ourselves accountable for the quality of the released product and provide you with the most relevant information.

On this occasion and on behalf of all QA team I’d like to say a big thank you to our active and loyal community for continuously sharing with us your experience with OXID eShop. Please continue to send feedback and help us improve the quality of our products even further!

Download your personal copy of OXID eShop Quality Report 4.6.0 here.

OXID eShop quality report

Some days ago we sent an internal quality report under confidentiality to our partners. After receiving a lot of requests to be allowed to publish this or parts of, we decided to publish the contents as it is. Yes, that’s right, the original confidential contents.

So what does report mean and how should you interpret this? Lets answer some questions:

For who was it written for? 
Well, frankly, the report was actually really only for internal use. So that management and other departments in OXID could get an update on the quality status. Then somebody had the bright idea that this may interest the partners. We did that and .. the rest is history as they say .. now it’s public 🙂

Who wrote it?
The Quality Manager, Dainius Bigelis, responsible for the eShop quality assurance.

What does it mean ?
Basically this report presents some statistics which are important for quality management of the OXID eShop.

Ok, so what’s in it, then?  

First, it’s about the source code in general. Here are some numbers about how large the code actual is, and how much of that is comments (more comments means better documented).


Why is source+comment <> total?. 
Because empty lines are counted in the total but its not separately listed here.

Why is the count different between PE and CE? Aren’t they 100% the same ? 
Main difference comes from license comments in each file. The CE file comment is larger then the PE. 

Especially interesting is the dispersion over time as shown here.




Unit testing: Source code coverage means how much percentage of the code is covered with unit tests. Lets see what we have:

Currently coverage of eShop source code by unit tests is:

  • EE 88.28
  • PE 90.59
  • CE 90.93


and it’s continuously rising. BTW, we use PHPUnit for the PHP unit testing. You will have code which is difficult to unit test. We are at the 90% border. Ask the unit tests experts, they will confirm that this is very good! We have now about 3000 unit tests per edition. 

As you can see the number of code in unit testing and selenium testing is growing consistently. Meaning better quality 🙂

Ok, the good is tested well. But is the code itself good maintainable and easy adaptable? You cannot tell that by looking at the size, or numbers of tests, can you? 
No, that’s why we also check the cyclomatic code complexity and style convention errors of our code. 


Cyclomatic complexity: This is a very interesting diagram! It gives a good indication if your code is to complex. How is it measured? Simple said, the more independent paths through a function, the more complex it is. See for more details. We defined 20 as to much. Actual experts arguing, that with PHP 30 to 35 is not that bad either. We decided to go for 20 firstly. Our software should also be understandable by developers who just want to use the eShop and don’t want to spend much time to read it. As you can see nicely in the diagram, we found that over 60% of our old version of eShop broke this rule and also that we quickly refactored it to much lower levels. During development it raised a bit but in the quality assurance phase it lowered again. So what about the last couple of % ? Some functions do have good reasons to be higher. 

Code conventions that important? Yes, they are, we want good readable code.


Bug tracking: 


Here you can see how many bugs we had, how many are fixed, which severity and status dispersion they have. The diagram below shows the bug count cumulative over time. The red line is the most interesting one, it shows how many opened bugs we have at any time. I think this speaks for itself.


So, what’s the conclusion ? 
I think we can safely say that the eShop code base has very good quality and have improved immensely compared to the previous version. Something to be proud of :-).