CI Through Project Life Cycle

The main goal of an IT project with destination for commercial market is to create a product from which its producer will benefit. To be competitive, the product needs to meet the following requirements:

• Quick time-to-market,
• Increase in customer satisfaction,
• Ensured adaptability of the delivered solution.

The above-mentioned criteria are equally applicable to projects focused on developing services, standalone applications, embedded software, etc. Meeting final customer expectations requires continuous development which leads to continuous integration. Different aspects of continuous integration have been described in “Modeling continuous integration practice differences in industry software development”. [1] Although the article written by Daniel Ståhl and Jan Bosch is rich source of information, the approach to software development and testing is constantly evolving to face a more and more demanding market.

In the beginning, there was chaos

First words describing the origins of the world in Greek Mythology can be applied to any IT project. In the beginning, there is an idea with a general definition of goals. Through time projects become more and more specified to finally reach a point when they’re close to an evolving living organism. Project life cycle can be divided into stages, as shown in Figure 1.
  

Figure 1. Project Life Cycle

For each project, life stage tests need to be focused on a different aspect to ensure that the final product meets customer requirements. How to implement Continuous Integration (CI) into project life cycle, so it would satisfy all needs of each development stage? To answer this question, it is necessary to first understand what activities should be performed at each stage and how can CI support it. 

A bunch of necessary definitions before diving into the project life cycle:

• Continuous Integration (CI) - a development practice focused on integrating each code change with a common code repository and executing a set of tests for verification purposes. The primary goal of CI is to detect issues by testing the smallest possible portions of code as often as possible.
• Shift left-  an approach which allows finding and preventing defects early in the software delivery process. Quality improvement is reached by moving tasks early in the lifecycle (moving left in the project timeline). Testing needs to be performed from the earliest phases of software development. [11]
• Test framework- mostly a set of guidelines for creating and designing test cases. It is a conceptual part of automated testing that helps testers to use resources more efficiently. 
• Tools – mechanisms (e.g., earned value computations) applied to inputs (e.g., task results) to create outputs (e.g., a progress report)
• Process – "a series of actions bringing about a result", while a result is a "concrete outcome."
• Project – a specific set of activities designed to accomplish a singular goal.  It has a defined beginning and end in time, and therefore limited scope and resources. [4,5]

CI in project life cycle

Based on Continuous Integration and Shift Left definitions, we can say that testing in a project needs to start as early as possible and must be performed even for the smallest changes.

Requirement Analysis

At this stage business requirements and a way to meet them are defined. Moreover, it is identified what products need to be delivered, what resources are needed and what set of skills is necessary for completing the work .

Although during the requirement analysis stage main work is performed by project managers, test architects and engineers should use this time for necessary analysis related to tools, frameworks, software development, and testing strategy. At some point, you need to decide whether to create Continuous Integration from scratch, or reuse a known solution. Tables below present both these approaches, as well as a combination of them.

Defining the CI chain with reuse of known solutions

Reuse of available tools, frameworks, and processes.

Advantages:

• Tools functionalities and limitations are well known.
• Tools and frameworks are integrated and ready to use.
• Development and test environment issues have been solved in an earlier project.
• Engineers are familiar with processes and way of work.

Disadvantages:

• Tool limitations may affect development and testing of functionalities mandatory for new projects.
• Available frameworks and procedures may not be scalable for new requirements.
• Tools and frameworks may have a high entry level for new employees.
• High costs of modernization in case of significant changes in specific project.

Recommendation:

One-to-one reuse of available resources (tools, frameworks, processes, and other methods) is useful when a new project is based on an already developed and maintained product – usually, a percentage of source code is inherited.

Defining the CI chain from scratch

Creating Continuous Integration based on new tools, frameworks, and process redefinition.

Advantages:

• Tools and frameworks can be selected to cover all requirements related to development and testing of project functionalities.
• Framework and defined processes suit project needs.
• Tools and frameworks can be chosen in a way which ensures a low entry level for unexperienced employees.

Disadvantages:

• Integration of tools and frameworks requires additional time for solving issues related to test and development environment.
• New processes and way of work require time and resources for introduction and successful deployment.
• The legacy test may not be reusable.

Recommendation:

This approach is applicable in two situations.

First: when you are starting a new project and there are no resources that could be reused.

Second: when you are starting a new platform as a base for further projects.

Because introduction and deployment of tools, frameworks, and processes requires significant time and resources, this approach is not recommended for projects with a challenging time-to-market.

Defining the CI chain with new tools and frameworks, as well as process redefinition

Partial reuse of tools, frameworks, and processes.

Advantages:

• Tools with limitations which do not allow for project functionality development and testing can be replaced within an existing framework.
• Legacy resources which meet project requirements can be reused to save time.
• Tools and frameworks with a high entry-level can be gradually replaced with more commonly used tools and frameworks.
• Issues with tool integration are limited to areas where changes have been made.

Disadvantages:

Integration of new tools and frameworks with available resources requires time for solving issues related to development or test environment.

Recommendation:

Due to reuse of some elements of the CI chain, it is possible to use this approach in a project with a medium to long time-to-market. Partial reuse is also possible in projects with short time-to-market, but you need to ensure that reused tools, frameworks, and processes don't have any dependencies on elements of the CI chain which you are not going to implement in the project CI.

The second activity in which engineers need to be involved is reviewing all available project related requirements. That way any limitations and bottlenecks will be discovered at the earliest stage possible and they can be properly addressed during project planning. Every time project related documentation is updated, a review session needs to be performed. Usage of static testing techniques like review, walkthrough, or inspection is quite effective when done by experienced engineers. Because static testing does not require test line or writing source code, it is also considered a cheapest way of detecting issues. 

Requirement analysis exit point 

At the end of the requirement analysis stage, you should have visibility of: project time-to-market, functionalities requested by customers, and similarities to already developed projects. Use this knowledge to predefine future Continuous Integration aspects, like:

• Potential for reuse 
• Risks related to the introduction of new tools, frameworks, and processes
• CI steps
• Speed and order in which CI steps need to be created and ready to use
• Main areas on which you will need to focus during further project development
• Type and number of needed equipment 

Although assumptions you will make at this stage may not be 100% accurate - they will be a good starting point for designing a phase where requirements towards Continuous Integration will be defined. 

Design

The Design stage is dedicated to defining solutions to meet requirements set by the customer. During this stage, software, hardware, and system architects decompose the assigned project into platforms and components. They define communication interfaces, hardware, and software procedures, as well as software and hardware components that are reused.

At this point, the project starts to become a product. For engineers responsible for designing and future implementation of the Continuous Integration chain, this is a crucial point. This is due to the fact that based on product architecture they define the complexity of CI, the function of each step, and acceptance criteria which need to be met on each stage - from committing to releasing.  

Figure 2. An adapted concept of  OpenEmbedded systems [2]

The system presented in Figure 2 has a layer structure like the ISO/OSI model, where an upper layer is creating services based on resources provided by lower layers. For that kind of architecture, Continuous Integration needs to ensure:

• A build system that allows creating software image with different versions of components on each layer.
• Independent development and verification of a single component within a single layer
• Independent integration of components within a single layer
• Functional and nonfunctional verification of a single layer 
• Integration of layers 

End-to-end verification of the whole system

Meeting all of the above criteria allows creating a multilevel CI chain, which thanks to testing performed from a single component level through layer integration to full system verification, ensures early issue detection and short feedback time. Functional, nonfunctional, structural, and regression tests apply to this approach. Implementation of any CI shell requires a set of tools. Those aspects are described in CI as a toolbox.

Testing in the design phase

In projects where both software and hardware are developed, it is a common practice to test potential solutions on a simulated target environment before the final implementation. This approach allows early detection of potential issues and choosing the most efficient solution without involvement of expensive equipment. This stage of design is often called the proof of concept phase. It is essential to remember that all defined tests and acceptance criteria must be appl for simulator-based test environments as well as for the target product. It saves time during the implementation phase.  

Design phase exit point

During this stage of project life cycle assumptions from the previous phase ha been refined and aligned with product architecture. Based on the specification provided by architects, Continuous Integration steps have been defined along with acceptance criteria and necessary tools. It’s time to start implementation phase.

Implementation

During this phase features requested by the customer are implemented into the final product. It starts development of software, and if needed hardware. To ensure proper quality of the final product, continuous practices need to be implement. Up till this point, work on the project related CI chain focused on analyzing requirements, tools, available frameworks, and designing a test chain to cover product architecture.

Although testing was limited to static methods and manual tests, each issue found in requirement analysis or design phase was cheap to correct. Rules related to testing as early as possible and integrating smallest possible elements have been fulfilled. Transition to the implementation phase of CI should be based on plans and designs prepared in earl phases. Along with source code development, all aspects of Continuous Integration evolve. It is essential to implement a version control system over source code and a build system at the beginning of product development – at the implementation phase. Having those two elements from the beginning allows more natural control of product development and integration into one image.

Testing aspect of CI needs correlating with the speed of product implementation. Tests for product functionalities should be available before implementation of the feature to main software line - trunk. Test-driven development (TDD) allows active issue detection by confronting code implementation with tests written based on the same requirements. Scope of tests executed within CI is changing together with product development. 

Verification on the component level may be possible only at the beginning. While the product is under development, next levels of testing defined during the project design phase can be enabled. Figuree 3 shows how Continuous Integration may evolve along with project development.

Figure 3. Project CI evolution during implementation phases

Implementation exit point

Project implementation stage ends when the product is delivered to the customer. From the technical point of view, it means that all required features have been turned into source code, have been tested, and have met acceptance criteria. Consecutive iterations of CI allow detecting and fixing issues in the developed product.

Maintenance

Maintenance - a phase, which is not in the scope of the Project Manager, but is essential for the customer and from the end user point of view. Well planned and implemented CI decreases the risk of corrections on faults revealed after delivering the final product. During maintenance more significant weight is placed on regression and finding issues related to exceptional border cases. From the customer point of view, at this stage of Project Life Cycle Continuous Integration especially the testing part is facing challenges of fast verification (with full regression) and delivering a correction to the customer.  

CI as a toolbox 

Continuous Integration as a process of automatic building and testing of code requires the use of various types of tools like a version control system, build system, automation server, test framework, and more. Remember that there is no such thing as one ultimate kit that always works for all projects, including yours. Consider the CI chain as a toolbox which needs to contain tools allowing you to get the job done. 

Before selecting a specific tool,  define a set of requirements that must be met by the potential solution - it helps to choos the best option for you. Below are some questions that may help you with selecting right tools for your CI:

• What is the entry level for users and administrators?
• Is it an out of the box solution or does it require an additional configuration and or update?
• Is It Open Source or Commercial?
• Is the tool commonly used or internal?
• What are purchase, deployment, and maintenance costs of the selected tool?
• Is there support available and what does it look like?
• Does the tool require a specific operating system (OS)?
• How much system resources does the tool require?
• Can it be run in a cloud environment?
• Can it be easily equipped with new functionalities? 
• How many specialists familiar with the specific tool are available on the market? 

It is critical not to get attached to one set of tools connected via case specific interfaces because at some point you will face a situation when the CI toolbox requires reorganization. If this happens at the begging of a project, tool replacement may be done relatively easily because the number of users is rather small. The situation becomes more complicated when you are doing a significant number of commits, the CI chain is working and can't be stopped. In this case, switching to a different tool takes some time, and it requires a transitional period when two solutions need to be supported. Before you swap tools in your CI, ensure that:

• The new solution meets all your requirements,
• The change does not cause any functional regression in CI,
• Potential time break in the CI chain is minimal,
• The new tool has potential to be further developed in the f,
• Added value (for example reduced test duration or ability to perform more complex test cases) is introduced in CI.

Although changing anything in CI is not as easy as replacing a screwdriver with a wrench, if you see that a tool has reached its limit and does not allow doing something more efficiently, you should replace it and move forward.

Conclusions

Continuous integration and all continuous practices have become an important area of software engineering research and practice, as well as a project management area of interest. There are developed approaches, tools, methods, and yet still CI needs to evolve to meet the need for delivering high-quality products while continually reducing time-to-market.

From The Authors

Dear Reader, thank you for reading this article. We hope that we have managed to bring you closer to the subject of Continuous Integration, especially how it has evolved from raw requirements into a system ensuring quality for a whole product. If you are interested in a deeper dive into CI, we strongly encourage you to study all materials from reference section.

About The Authors:

Grzegorz Czaiński:

I started work in IT in 2010 as an Integration and Verification Engineer. During my career in Nokia, I've gained experience as a tester, Integration Leader, Product and Process Owner. Currently, as a Verification Architect, I'm responsible for defining, implementing,  tracking processes and practices related to Test Strategy at the department level.

Tatsiana Viarbitskaya:

I started my career in Nokia in 2015. Now I work as a Product Owner in MN SR SW R&D WRO Architecture Team, where we are creating Verification Architecture Core. I, also actively participate in various international and national conferences about signal processing.

References:

[1] Ståhl, D and Bosch, J "Modeling continuous integration practice differences in industry software development," Journal of Systems and Software, vol. 87, pp. 48-59, 1, 2014.

[2] Peregudov, O. (2018) RFSW Archeo RoadMap (adapted view), pp. 51-53, 2018

A Guide to the Project Management Body of Knowledge (PMBOK® Guide ver 6th)

Duncan, W. R. (1993). The process of project management. Project Management Journal, ISSN: 1938-9507, 24(3), 5–10.

Minnesota Office of Continuous Improvement, [Online]. Available: https://mn.gov/admin/continuous-improvement/resources/projects/toolbox/

Shahin, M., Babar, A, Zhu, L, "Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges, and Practices," IEEE Access, ISSN 2169-3536, pp. 3909 – 3943, 2017.