Update Jenkins plugins behind a corporate proxy

Reading Time: 4 minutes

Many teams are running Jenkins in their environment. Most of the time it needs to go through a corporate proxy in order to access external resources. It’s relatively easy to configure a proxy in the advanced plugin configuration. But if each domain needs to be white-listed, trouble starts almost certainly.

But let’s start with possible ways of updating Jenkins plugins.

The offline aka manual scenario

If you aren’t allowed to communicate with the internet from within your productive Jenkins environment at all, you still have some options to choose from:

  1. Don’t update
    Not a good idea. Period.
  2. Manually check and download all your plugins with the plugin index https://plugins.jenkins.io/
    Obviously, you don’t want to do this for a typical Jenkins setup with likely more than 70 plugins.
  3. Run a second Jenkins in a zone where you DO have Internet access
    Check for updates and download the desired plugins. Then copy all the .hpi files to your productive Jenkins.

The corporate proxy scenario

If you’re allowed to communicate through a corporate proxy with the Internet, your half-way. But many corporate proxies force you to white-list all required domains or IP’s in order to control access to external resources.

At first, you might think “yeaahhhh… no problem!” and you ask your network security colleagues to enable:

But then, the Jenkins mirroring features hits you.

The meta-data JSON https://updates.jenkins.io/update-center.json provides binary URLs like this: http://updates.jenkins-ci.org/download/plugins/slack/2.10/slack.hpi

But if you call this URL, you get first redirected to http://mirrors.jenkins-ci.org/plugins/slack/2.10/slack.hpi and then another redirect is done depending on your geographic location. In my case, I get redirected to http://ftp-chi.osuosl.org/pub/jenkins/plugins/slack/2.10/slack.hpi

As the status of the mirrors might change, the possibility of returned domains change as well and you’ll find yourself talking to your network security guy quite often.

By-pass mirroring by rewriting meta-data JSON

One possible solution to go round this mirroring feature is to download the update-center.json, rewrite all links and then use the rewritten JSON.

Download and rewrite JSON

Downloading and rewriting the official update-center.json could be done with many technologies. But I choose Jenkins for this as well. Therefore I created a simple Jenkins job name “update-center”, which is scheduled to run once every day.

The following declarative pipeline does the job:

pipeline {
  agent any
  stages {
    stage('Download and modify Update-Center Data') {
      steps {
        httpRequest(
                url: "https://updates.jenkins.io/update-center.json?id=default&version=" + Jenkins.instance.version,
                consoleLogResponseBody: false,
                acceptType: 'APPLICATION_JSON',
                httpProxy: 'http://my.corporate.proxy:8080',
                outputFile: 'update-center.json'
        )
        script {
          updateCenterJson = readFile file: 'update-center.json'
          updateCenterJson = updateCenterJson.replaceAll("http:\\/\\/updates\\.jenkins-ci\\.org\\/download\\/", "http://archives.jenkins-ci.org/")
        }
        writeFile text: updateCenterJson, file: 'update-center-updated.json'
        archiveArtifacts 'update-center-updated.json'
      }
    }
  } 
}

Some notes regarding the above pipeline:

  • You need to replace my.corporate.proxy:8080 with your actual proxy.
  • We read the current installed Jenkins version with Jenkins.instance.version. This needs to be explicitly approved: https://jenkins.io/doc/book/managing/script-approval/. If this isn’t an option, the version has to be hard-coded.
  • The https://plugins.jenkins.io/http_request plugin is used to download the JSON. You could achieve a similar thing with a simple curl if you don’t want this plugin.
  • You still need to white-list those two domains in your corporate proxy:
    • https://updates.jenkins.io
    • http://archives.jenkins-ci.org
  • Instead of using archives.jenkins-ci.org, you should use a mirror as the official archives server doesn’t provide great performance.

Use the rewritten JSON

Proxy configuration

Go to the advanced Plugin Manager configuration http://localhost:8080/pluginManager/advanced (or Jenkins > Manage Jenkins > Manage Plugins > Advanced) and configure your corporate proxy:

Update Site configuration

You can configure the URL to the JSON at the bottom of the same page.

In my setup, http://localhost/job/update-center/lastSuccessfulBuild/artifact/update-center-updated.json is used.

Now the update will check and download behind your corporate proxy!

Questions?

Should you have specific questions on this subject, please feel free to comment and share any question you have. If it helped you please like ❤️ or share📌 this story, so others like you can find it.

Follow N47 on Instagram, Twitter, LinkedIn, Facebook to get updated as soon as possible.

28 replies
  1. Pierre
    Pierre says:

    Hi,

    Very interesting and nice article but in my case (Jenkins 2.150.3) it is not working because of some signature issue. Indeed, everytime I try to check updates, I have an error message on “SHA-512 digest mismatch” that is displayed. I tried to change the java.security file in Jenkins JRE directory but I’m not able to work around this one. Any suggestion ?

    Thanks

    Reply
      • Snipe
        Snipe says:

        Hi Stefan,

        I tried to use your approach with the rewriting of the URLs in order to have a fixed “mirror”. However, I ran into the same error as Pierre. It seems it has nothing to do with SSL check. It seems that Jenkins is checking the signature (certificates, correct_digest, correct_digest512) which are generated once the json file is generated. When you modify the json file in any way you would need to re-sign the json file. Is this really working for you? Which version of Jenkins are you using?

        Reply
      • Stefan Maurer
        Stefan Maurer says:

        Hi Varinder

        Happy to hear you found a solution. And thanks a lot for sharing it.

        It worked for me as described with Jenkins version 2.160.

        Reply
        • Snipe
          Snipe says:

          Hi Stefan,

          it seems to work. One error that I needed to fix was that the resulting file must be named “update-center.json”, otherwise it won’t work even with signature check turned off (Message: “There were errors checking the update sites: The update site null does not look like an update center”). However – I am not willing to use this on corporate production server – after all the signature check has a purpose. We decided to open the firewall for the mirrors instead.

          Reply
  2. Switi
    Switi says:

    I am getting error in my jenkins as Signature verification failed in update site . java.security.cert.CertificateExpiredException:Not After Tue 19 . Unable to install plugins.Any suggestion how to sort out issue?

    Reply
    • Stefan Maurer
      Stefan Maurer says:

      Have you manually imported root certificates in the Trust Store as proposed in my comment from 18/06/2019 at 13:34? They might got expired and need to get imported again.

      Reply
      • Switi
        Switi says:

        Thanks. Issue got resolved by updating json file.
        Root cause: Issue was in older json file and proxy also got decommission which using in Jenkins.

        Reply
          • Switi
            Switi says:

            Hi Stefan..After upgrading Jenkins version 2.204.1 config file not showing in the managed files. Can you help on this?

          • Stefan Maurer
            Stefan Maurer says:

            Hi Switi

            We reached the max depth of comments 🙂

            Sorry, I don’t get your question. What do you mean by “managed files”?

  3. honglus
    honglus says:

    Following is alternative solution to setup python:SimpleHTTPServer
    It is suitable for docker based Jenkins.
    if you use Kubernetes, put following commands in lifecycle/postStart in container creation.

    cd /var/jenkins_home
    jenkins_version=$(sed -n -e ‘s|\([0-9].*\)|\1|p’ config.xml | sed -e ‘s/ //g’)

    curl -L -o update-center.json -q “https://updates.jenkins.io/update-center.json?id=default&version=$jenkins_version”

    sed -i ‘s|http://updates.jenkins-ci.org/download|http://archives.jenkins-ci.org/|g’ update-center.json
    nohup python -m SimpleHTTPServer 8888 > /dev/null

    then replace https://updates.jenkins.io/update-center.json with ‘http://localhost:8888/update-center.json’ in Jenkins console or baked in docker image

    Reply
    • honglus
      honglus says:

      just realized the special char was removed when published and it is security risk to expose /var/jenkins_home. go to /tmp/ instead
      cd /tmp
      jenkins_version=$(sed -n -e ‘s|.version.\([0-9].*\)./version.|\1|p’ config.xml | sed -e ‘s/ //g’)

      Reply
  4. Switi
    Switi says:

    Hi All,
    Am facing issue as no valid crumb was included in the request after upgrade jenkins version.
    Anyone have any idea??

    Reply
    • Stefan Maurer
      Stefan Maurer says:

      Hi Madhu, Gianfranco and Yves

      Please check the permissions of the job which pulls the update site. You might need to give Read access to anonymous users for that specific job:
      https://wiki.jenkins-ci.org/display/JENKINS/Standard+Security+Setup

      An alternative could be as well the suggestion from honglus to use s simple HTTP Server instead of Jenkins: https://www.north-47.com/knowledge-base/update-jenkins-plugins-behind-a-corporate-proxy/#comment-29

      Hope that helps.
      Stefan

      Reply
      • Yves
        Yves says:

        Thanks a lot , Stefan.
        This and the comments posted before helped a lot indeed.
        I managed to setup the custom update site (in Jenkins 2.235).
        Here’s a summary of the possible errors, causes and solutions:

        Using “localhost” in the URL gave “Server returned HTTP response code: 503” because we’re not running Jenkins locally.

        We replaced “localhost” by the name of the server where our Jenkins instance runs
        We then received “Server returned HTTP response code: 403” because anonymous users need to be able to read the project’s output file => (you need the plugin “Matrix Authorization Strategy Plugin”)

        Manage Jenkins => Configure Global Security => Project-based Matrix Authorization Strategy: check box “Overall” “Read” for “Anonymous users”
        => the error became “file not found”

        We still needed to grant that permission at project level, i.e. configure the project that creates the file: check box “Job” “Read” for “Anonymous users”
        => the error became “SHA-512 digest mismatch”, showing that the new file is not signed.

        We disabled this by adding JAVA option argument
        -Dhudson.model.DownloadService.noSignatureCheck=true
        => the error became “The update site null does not look like an update center”

        In the pipeline job, we set the name of the output file to “update-center.json” instead of “update-center-updated.json”.
        => No more error !

        I hope this can help others.

        Reply
        • Stefan Maurer
          Stefan Maurer says:

          Hi Yves

          I’m glad my comment was helpful.

          And thanks a lot for sharing further issues (together with the solutions).

          Cheers,
          Stefan

          Reply
        • Gianfranco
          Gianfranco says:

          Hi Yves, I followed your suggestions and fixed the HTTP 403 and SHA-512 digest mismatch problems, but now I have another problem: There were errors checking the update sites: The update site default does not look like an update center. Has this ever happened to you?

          Reply
    • Stefan Maurer
      Stefan Maurer says:

      Hi Yves

      Please read my above comment regarding 403.

      Regarding the URLs. Jenkins might have changed that.

      Maybe you could use as well this code in order to avoid an extra redirect:


      pipeline {
      agent any
      stages {
      stage('Download and modify Update-Center Data') {
      steps {
      httpRequest(
      url: "http://updates.jenkins.io/dynamic-stable-" + Jenkins.instance.version + "/update-center.json",
      consoleLogResponseBody: false,
      acceptType: 'APPLICATION_JSON',
      httpProxy: 'http://my.corporate.proxy:8080',
      outputFile: 'update-center.json'
      )
      script {
      updateCenterJson = readFile file: 'update-center.json'
      updateCenterJson = updateCenterJson.replaceAll("http:\\/\\/updates\\.jenkins-ci\\.org\\/download\\/", "http://archives.jenkins-ci.org/")
      }
      writeFile text: updateCenterJson, file: 'update-center-updated.json'
      archiveArtifacts 'update-center-updated.json'
      }
      }
      }
      }

      But please note, that I could verify/test this approach.

      BR
      Stefan

      Reply

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *