What is CI? Continuous Integration Explained

Reading Time: 5 minutes

Continuous Integration (CI) is a software development practice that requires members of a team, to frequently integrate their code changes into a central repository (master branch), preferably several times a day.

Each merge is then verified by automatically generating a build, and running automated tests against that build.

By integrating regularly, you can detect errors quickly, as well as locate and fix them easier.

Why is Continuous Integration Needed?

Back in the days, BCI – Before Continuous Integration, developers from a single team might have worked in isolation for a longer period of time, and they merged their code changes only when they finished working on a particular feature or bug fix.

This caused the well-known merge hell (integration hell) or in other words a lot of code conflicts, bugs introduced, lots of time invested into the analysis, as well as frustrated developers and project managers.

All these ingredients made it harder to deliver updates and value to the customers on time.

How does Continuous Integration Work?

Continuous Integration as a software development practice entails two components: automation and cultural.

The cultural component focuses on the principle of frequent integrations of your code changes to the mainline of the central repository, using a version control system such as Git, Mercurial or Subversion.

But applying the cultural component you will drastically lower the frustrations and time wasted merging code, because, in reality, you are merging small changes all the time.

As a matter of fact, you can practice Continuous Integration using only this principle, but with adding the automation component into your CI process you can exploit the full potential of the Continuous Integration principle.

Continuous Integration Image
Source

As shown in the picture above, this includes a CI server that will generate builds automatically, run automated tests against those builds and notify (or alert) the team members of the results.

By leveraging the automation component you will immediately be aware of any errors, thus allowing the team to fix them fast and without too much time spent analysing.

There are plenty of CI tools out there that you can choose from, but the most common are: Jenkins, CircleCI, GitHub Actions, Bitbucket Pipelines etc.

Continuous Integration Best Practices and Benefits

Everyone should commit to the mainline daily

By doing frequent commits and integrations, developers let other developers know about the changes they’ve done, so passive communication is being maintained.

Other benefits that come with developers integrating multiple times a day:

  • integration hell is drastically reduced
  • conflicts are easily resolved as not much has changed in the meantime
  • errors are quickly detected

The builds should be automated and fast

Given the fact several integrations will be done daily, automating the CI Pipeline is crucial to improving developer productivity as it leads to less manual work and faster detection of errors and bugs.

Another important aspect of the automated build is optimising its execution speed and make it as fast as possible as this enables faster feedback and leads to more satisfied developers and customers.

Everyone should know what’s happening with the system

Given Continuous Integration is all about communication, a good practice is to inform each team member of the status of the repository.

In other words, whenever a merge is made, thus a build is triggered, each team member should be notified of the merge as well as the results of the build itself.

To notify all team members or stakeholders, use your imagination, though email is the most common channel, but you can leverage SMS, integrate your CI server with communication platforms like Slack, Microsoft Teams, Webex etc.

Test Driven Development

Test Driven Development (TDD) is a software development approach relying on the principle of writing tests before writing the actual code. What TDD offers as a value in general, is improved test coverage and an even better understanding of the system requirements.

But, put those together, Continuous Integration and TDD, and you will get a lot more trust and comfort in the CI Pipelines as every new feature or bug fix will be shipped with even better test coverage.

Test Driven Development also inspires a cultural change into the team and even the whole organisation, by motivating the developers to write even better and more robust test cases.

Pull requests and code review

A big portion of the software development teams nowadays, practice pull request and code review workflow.

A pull request is typically created whenever a developer is ready to merge new code changes into the mainline, making the pull request perfect for triggering the CI Pipeline.

Usually, additional manual approval is required after a successful build, where other developers review the new code, make suggestions and approve or deny the pull request. This final step brings additional value such as knowledge sharing and an additional layer of communication between the team members.

Summary

Building software solutions in a multi-developer team are as complex as it was five, ten or even twenty years ago if you are not using the right tools and exercise the right practices and principles, and Continuous Integration is definitely one of them.


I hope you enjoyed this article and you are not leaving empty-handed.
Feel free to leave a comment. 😀

Follow N47 on InstagramTwitterLinkedInFacebook for any updates.

Project story: Automate AEM deployments for a Swiss bank

Reading Time: 5 minutes

A large bank in St. Gallen, Switzerland had the need to improve the AEM deployment process for its various staging environments. It was one of my first projects for N47 and was settled to run for 6 months starting in October 2018. The following blog gives a short project overview.

Getting started

Starting to work for a new customer is always exciting to me because every company and team has a unique mindset and culture. Usually, it takes a few days or weeks to get to know the new teammates. But this time it was completely different, as I already worked with each of the three team members and their supervisor together in one before my previous company. It was nice to meet old colleagues and we had a very good start.

Deployment process before automation.
Source: www.dreamstime.com

Technology Stack

The technology stack was already defined and the servers ordered. But it took a while until the infrastructure was ready and for the time being, I worked on my local machine.

Jenkins was set as the central tool for build orchestration, deployments, and various DevOps tasks. All the pipeline source code is stored in GitLab and the main business application we’re dealing with is Adobe Experience Manager (AEM).

A relatively large amount of work was needed for the initial setup like enabling connectivity to the relevant systems, basic shared library, and getting to know the internal processes. Read more about Jenkins behind a corporate proxy as an example for this setup: https://www.north-47.com/knowledge-base/update-jenkins-plugins-behind-a-corporate-proxy/

Implemented Pipelines

The bank has two different AEM projects: one for the corporate website and another for their intranet. They require a slightly different deployment pipeline and both have three environments: development, staging, and production.

Besides the deployment pipelines, there are pipelines for copying content from the production to the development environment and restoring a complete production environment into the staging environment in order to have an exact copy and a good baseline for approvals.

Many auxiliary jobs like start/stop AEM + Dispatcher, checking the health of instances, fetch last backup time, and execute Groovy scripts are used in the deployment pipelines as well in an independent executable job.

An example of a Jenkins Pipeline
Source: https://jenkins.io
An example of a Gitlab Pipeline
Source: https://docs.gitlab.com/ee/ci/pipelines.html

Advantages of automation

The automation of the various processes brought faster deployments. But more important transparency and centralized logs about what exactly happened and higher quality as repetitive tasks are always executed.

One example is the backup check, which needed coordination and forced to long waiting times. Now an API is used and the automation pipeline has instant feedback about the last backup time and shows a note if a backup is missing. Before, such a step might have been skipped in order to save some time.

With each built pipeline, some more little and reusable helpers were introduced which made it then again easier and faster to create the next pipelines. Think of a construction kit.

Deployment process with automation.
Source: www.wikimedia.org

Project finished ➞ client is very happy

After several months of close collaboration, more and more pipelines have been implemented and are used to support the crucial deployment processes countless times.

I enjoyed building-up the AEM automation and believe it’s a very good aid for higher quality and further extensions.

After a warm welcome and six months of working together, it was time to say good-bye as the project had a fixed time span. The client’s team was very kind and gave me even some great presents to remember the exciting time in St. Gallen.

Present from client: Swiss beer, chocolate, bratwurst and biber

JPoint Java conference in Moscow – 2019

Reading Time: 2 minutes

JPoint is one of the three (JPoint, Joker and JBreak) most common technical Java conferences for experienced Java developers. There will be 40 talks in two days, separated in 4 tracks in parallel. The conference takes place each year, this is being the seventh consecutive year.

Organization to visit JPoint conference

Apart from changing flights to reach Moscow everything else should not be any bigger issue. Book flights and choose some nearby hotel.

There are a few types of tickets. From which I’ll choose the personal ticket, main reason is the discount of 58%.

What is scheduled by now?

Many interesting subjects are going to be covered during two days of presentations:

  • New projects in Jenkins
  • Java SE 10 variable types
  • More of Java collections
  • Decomposing Java applications
  • AOT Java compilation
  • Java vulnerability
  • Prepare Java Enterprise application for production
  • Application migration JDK 9, 10, 11 and 12
  • Jenkins X

The following topics on the conference will be the most interesting ones for me:

  • Prepare Enterprise application for production (telemetry is crucial).
  • Is Java so vulnerable? What can we do to reduce security issues?
  • What is the right way of splitting application to useful components?
  • It looks that now with Jenkins Essentials there is significant less overhead for managing it, without any user involvement. Let us see what Jenkins replaced with few commands.

Just half of presentations are scheduled by now. Expect many more to be announce.

My expectations on JPoint Moscow 2019

Reading Time: 3 minutes

PREPARATION

Tickets

Tickets for individuals: 280€ until 1st March.
No possibility to change the participant.

Personal tickets may not be acquired by companies in any way. The companies may not fully or partially reimburse these tickets’ costs to their employees.

Standard tickets: 465€ until 1st March. A possibility to change the participant is given.

Tickets for companies and individuals, no limits. Includes a set of closing documents and amendments to the contract.

Flight

Skopje-Vienna-Moscow. Visa for Russia is needed!

Hotel

I guess a hotel like Crowne Plaza Moscow – World Trade Centre is a good option, because it’s in the same place where the conference takes part.

DAY 1

So what are my plans and expectations for the first day of JPoint. I will start with Rafael Winterhalter who is a Java Champion and will talk about Java agents. It will be interesting to see how Java classes can be used as templates for implementing highly performant code changes.

Next stop will be the creator of Jenkins: Kohsuke Kawaguchi. He has great headline Superpowers coming to your Jenkins and I am exicted to see where Jenkins is going next.

Last stop for day one, Simon Ritter from Azul Systems, with focus on local variable type inference. As with many features, there are some unexpected nuances as well as both good and bad use cases that will be covered.

There will be many more for day one but I will focus on these three for now. Also at the end, party at 20:00.

DAY 2

I will start the second day with Simon Ritter again, this time with focus on JDK 12. Pitfalls for the unwary, it will be interesting to see all the areas of JDK 9, 10, 11 and 12 that may impact application migration. Another topic will be how the new JDK release cadence will impact Java support and the choices of which Java versions to use in production.

Other headliner talks for the second day are still under consideration, so I’m expecting something interesting from Pivotal and JetBrains.

Feel free to share some Moscow hints or interesting talks that I’m missing.

Update Jenkins plugins behind a corporate proxy

Reading Time: 5 minutes

Many teams are running Jenkins in their environment. Most of the time it needs to go through a corporate proxy in order to access external resources. It’s relatively easy to configure a proxy in the advanced plugin configuration. But if each domain needs to be white-listed, trouble starts almost certainly.

But let’s start with possible ways of updating Jenkins plugins.

The offline aka manual scenario

If you aren’t allowed to communicate with the internet from within your productive Jenkins environment at all, you still have some options to choose from:

  1. Don’t update
    Not a good idea. Period.
  2. Manually check and download all your plugins with the plugin index https://plugins.jenkins.io/
    Obviously, you don’t want to do this for a typical Jenkins setup with likely more than 70 plugins.
  3. Run a second Jenkins in a zone where you DO have Internet access
    Check for updates and download the desired plugins. Then copy all the .hpi files to your productive Jenkins.

The corporate proxy scenario

If you’re allowed to communicate through a corporate proxy with the Internet, your half-way. But many corporate proxies force you to white-list all required domains or IP’s in order to control access to external resources.

At first, you might think “yeaahhhh… no problem!” and you ask your network security colleagues to enable:

But then, the Jenkins mirroring features hits you.

The meta-data JSON https://updates.jenkins.io/update-center.json provides binary URLs like this: http://updates.jenkins-ci.org/download/plugins/slack/2.10/slack.hpi

But if you call this URL, you get first redirected to http://mirrors.jenkins-ci.org/plugins/slack/2.10/slack.hpi and then another redirect is done depending on your geographic location. In my case, I get redirected to http://ftp-chi.osuosl.org/pub/jenkins/plugins/slack/2.10/slack.hpi

As the status of the mirrors might change, the possibility of returned domains change as well and you’ll find yourself talking to your network security guy quite often.

By-pass mirroring by rewriting meta-data JSON

One possible solution to go round this mirroring feature is to download the update-center.json, rewrite all links and then use the rewritten JSON.

Download and rewrite JSON

Downloading and rewriting the official update-center.json could be done with many technologies. But I choose Jenkins for this as well. Therefore I created a simple Jenkins job name “update-center”, which is scheduled to run once every day.

The following declarative pipeline does the job:

pipeline {
  agent any
  stages {
    stage('Download and modify Update-Center Data') {
      steps {
        httpRequest(
                url: "https://updates.jenkins.io/update-center.json?id=default&version=" + Jenkins.instance.version,
                consoleLogResponseBody: false,
                acceptType: 'APPLICATION_JSON',
                httpProxy: 'http://my.corporate.proxy:8080',
                outputFile: 'update-center-original.json'
        )
        script {
          updateCenterJson = readFile file: 'update-center-original.json'
          updateCenterJson = updateCenterJson.replaceAll("http:\\/\\/updates\\.jenkins-ci\\.org\\/download\\/", "http://archives.jenkins-ci.org/")
        }
        writeFile text: updateCenterJson, file: 'update-center.json'
        archiveArtifacts 'update-center.json'
      }
    }
  } 
}

Some notes regarding the above pipeline:

  • You need to replace my.corporate.proxy:8080 with your actual proxy.
  • We read the current installed Jenkins version with Jenkins.instance.version. This needs to be explicitly approved: https://jenkins.io/doc/book/managing/script-approval/. If this isn’t an option, the version has to be hard-coded.
  • The https://plugins.jenkins.io/http_request plugin is used to download the JSON. You could achieve a similar thing with a simple curl if you don’t want this plugin.
  • You still need to white-list those two domains in your corporate proxy:
    • https://updates.jenkins.io
    • http://archives.jenkins-ci.org
  • Instead of using archives.jenkins-ci.org, you should use a mirror as the official archives server doesn’t provide great performance.

Use the rewritten JSON

Proxy configuration

Go to the advanced Plugin Manager configuration http://localhost:8080/pluginManager/advanced (or Jenkins > Manage Jenkins > Manage Plugins > Advanced) and configure your corporate proxy:

Update Site configuration

You can configure the URL to the JSON at the bottom of the same page.

In my setup, http://localhost/job/update-center/lastSuccessfulBuild/artifact/update-center.json is used.

Now the update will check and download behind your corporate proxy!

Permission

Note: You need to ensure, that anonymous users have read permission to your artifacts. Read more about security configuration here: https://wiki.jenkins-ci.org/display/JENKINS/Standard+Security+Setup

Questions?

Should you have specific questions on this subject, please feel free to comment and share any question you have. If it helped you please like ❤️ or share📌 this story, so others like you can find it.

Follow N47 on Instagram, Twitter, LinkedIn, Facebook to get updated as soon as possible.