Update Jenkins plugins behind a corporate proxy

How Can We Help?

Back
Print
Reading Time: 5 minutes

Many teams are running Jenkins in their environment. Most of the time it needs to go through a corporate proxy in order to access external resources. It’s relatively easy to configure a proxy in the advanced plugin configuration. But if each domain needs to be white-listed, trouble starts almost certainly.

But let’s start with possible ways of updating Jenkins plugins.

The offline aka manual scenario

If you aren’t allowed to communicate with the internet from within your productive Jenkins environment at all, you still have some options to choose from:

  1. Don’t update
    Not a good idea. Period.
  2. Manually check and download all your plugins with the plugin index https://plugins.jenkins.io/
    Obviously, you don’t want to do this for a typical Jenkins setup with likely more than 70 plugins.
  3. Run a second Jenkins in a zone where you DO have Internet access
    Check for updates and download the desired plugins. Then copy all the .hpi files to your productive Jenkins.

The corporate proxy scenario

If you’re allowed to communicate through a corporate proxy with the Internet, your half-way. But many corporate proxies force you to white-list all required domains or IP’s in order to control access to external resources.

At first, you might think “yeaahhhh… no problem!” and you ask your network security colleagues to enable:

But then, the Jenkins mirroring features hits you.

The meta-data JSON https://updates.jenkins.io/update-center.json provides binary URLs like this: http://updates.jenkins-ci.org/download/plugins/slack/2.10/slack.hpi

But if you call this URL, you get first redirected to http://mirrors.jenkins-ci.org/plugins/slack/2.10/slack.hpi and then another redirect is done depending on your geographic location. In my case, I get redirected to http://ftp-chi.osuosl.org/pub/jenkins/plugins/slack/2.10/slack.hpi

As the status of the mirrors might change, the possibility of returned domains change as well and you’ll find yourself talking to your network security guy quite often.

By-pass mirroring by rewriting meta-data JSON

One possible solution to go round this mirroring feature is to download the update-center.json, rewrite all links and then use the rewritten JSON.

Download and rewrite JSON

Downloading and rewriting the official update-center.json could be done with many technologies. But I choose Jenkins for this as well. Therefore I created a simple Jenkins job name “update-center”, which is scheduled to run once every day.

The following declarative pipeline does the job:

pipeline {
  agent any
  stages {
    stage('Download and modify Update-Center Data') {
      steps {
        httpRequest(
                url: "https://updates.jenkins.io/update-center.json?id=default&version=" + Jenkins.instance.version,
                consoleLogResponseBody: false,
                acceptType: 'APPLICATION_JSON',
                httpProxy: 'http://my.corporate.proxy:8080',
                outputFile: 'update-center-original.json'
        )
        script {
          updateCenterJson = readFile file: 'update-center-original.json'
          updateCenterJson = updateCenterJson.replaceAll("http:\\/\\/updates\\.jenkins-ci\\.org\\/download\\/", "http://archives.jenkins-ci.org/")
        }
        writeFile text: updateCenterJson, file: 'update-center.json'
        archiveArtifacts 'update-center.json'
      }
    }
  } 
}

Some notes regarding the above pipeline:

  • You need to replace my.corporate.proxy:8080 with your actual proxy.
  • We read the current installed Jenkins version with Jenkins.instance.version. This needs to be explicitly approved: https://jenkins.io/doc/book/managing/script-approval/. If this isn’t an option, the version has to be hard-coded.
  • The https://plugins.jenkins.io/http_request plugin is used to download the JSON. You could achieve a similar thing with a simple curl if you don’t want this plugin.
  • You still need to white-list those two domains in your corporate proxy:
    • https://updates.jenkins.io
    • http://archives.jenkins-ci.org
  • Instead of using archives.jenkins-ci.org, you should use a mirror as the official archives server doesn’t provide great performance.

Use the rewritten JSON

Proxy configuration

Go to the advanced Plugin Manager configuration http://localhost:8080/pluginManager/advanced (or Jenkins > Manage Jenkins > Manage Plugins > Advanced) and configure your corporate proxy:

Update Site configuration

You can configure the URL to the JSON at the bottom of the same page.

In my setup, http://localhost/job/update-center/lastSuccessfulBuild/artifact/update-center.json is used.

Now the update will check and download behind your corporate proxy!

Permission

Note: You need to ensure, that anonymous users have read permission to your artifacts. Read more about security configuration here: https://wiki.jenkins-ci.org/display/JENKINS/Standard+Security+Setup

Questions?

Should you have specific questions on this subject, please feel free to comment and share any question you have. If it helped you please like ❤️ or share📌 this story, so others like you can find it.

Follow N47 on Instagram, Twitter, LinkedIn, Facebook to get updated as soon as possible.

Table of Contents
45 replies
  1. Pierre
    Pierre says:

    Hi,

    Very interesting and nice article but in my case (Jenkins 2.150.3) it is not working because of some signature issue. Indeed, everytime I try to check updates, I have an error message on “SHA-512 digest mismatch” that is displayed. I tried to change the java.security file in Jenkins JRE directory but I’m not able to work around this one. Any suggestion ?

    Thanks

    Reply
      • Snipe
        Snipe says:

        Hi Stefan,

        I tried to use your approach with the rewriting of the URLs in order to have a fixed “mirror”. However, I ran into the same error as Pierre. It seems it has nothing to do with SSL check. It seems that Jenkins is checking the signature (certificates, correct_digest, correct_digest512) which are generated once the json file is generated. When you modify the json file in any way you would need to re-sign the json file. Is this really working for you? Which version of Jenkins are you using?

        Reply
      • Stefan Maurer
        Stefan Maurer says:

        Hi Varinder

        Happy to hear you found a solution. And thanks a lot for sharing it.

        It worked for me as described with Jenkins version 2.160.

        Reply
        • Snipe
          Snipe says:

          Hi Stefan,

          it seems to work. One error that I needed to fix was that the resulting file must be named “update-center.json”, otherwise it won’t work even with signature check turned off (Message: “There were errors checking the update sites: The update site null does not look like an update center”). However – I am not willing to use this on corporate production server – after all the signature check has a purpose. We decided to open the firewall for the mirrors instead.

          Reply
  2. Switi
    Switi says:

    I am getting error in my jenkins as Signature verification failed in update site . java.security.cert.CertificateExpiredException:Not After Tue 19 . Unable to install plugins.Any suggestion how to sort out issue?

    Reply
    • Stefan Maurer
      Stefan Maurer says:

      Have you manually imported root certificates in the Trust Store as proposed in my comment from 18/06/2019 at 13:34? They might got expired and need to get imported again.

      Reply
      • Switi
        Switi says:

        Thanks. Issue got resolved by updating json file.
        Root cause: Issue was in older json file and proxy also got decommission which using in Jenkins.

        Reply
          • Switi
            Switi says:

            Hi Stefan..After upgrading Jenkins version 2.204.1 config file not showing in the managed files. Can you help on this?

          • Stefan Maurer
            Stefan Maurer says:

            Hi Switi

            We reached the max depth of comments 🙂

            Sorry, I don’t get your question. What do you mean by “managed files”?

  3. honglus
    honglus says:

    Following is alternative solution to setup python:SimpleHTTPServer
    It is suitable for docker based Jenkins.
    if you use Kubernetes, put following commands in lifecycle/postStart in container creation.

    cd /var/jenkins_home
    jenkins_version=$(sed -n -e ‘s|\([0-9].*\)|\1|p’ config.xml | sed -e ‘s/ //g’)

    curl -L -o update-center.json -q “https://updates.jenkins.io/update-center.json?id=default&version=$jenkins_version”

    sed -i ‘s|http://updates.jenkins-ci.org/download|http://archives.jenkins-ci.org/|g’ update-center.json
    nohup python -m SimpleHTTPServer 8888 > /dev/null

    then replace https://updates.jenkins.io/update-center.json with ‘http://localhost:8888/update-center.json’ in Jenkins console or baked in docker image

    Reply
    • honglus
      honglus says:

      just realized the special char was removed when published and it is security risk to expose /var/jenkins_home. go to /tmp/ instead
      cd /tmp
      jenkins_version=$(sed -n -e ‘s|.version.\([0-9].*\)./version.|\1|p’ config.xml | sed -e ‘s/ //g’)

      Reply
  4. Switi
    Switi says:

    Hi All,
    Am facing issue as no valid crumb was included in the request after upgrade jenkins version.
    Anyone have any idea??

    Reply
    • Stefan Maurer
      Stefan Maurer says:

      Hi Madhu, Gianfranco and Yves

      Please check the permissions of the job which pulls the update site. You might need to give Read access to anonymous users for that specific job:
      https://wiki.jenkins-ci.org/display/JENKINS/Standard+Security+Setup

      An alternative could be as well the suggestion from honglus to use s simple HTTP Server instead of Jenkins: https://www.north-47.com/knowledge-base/update-jenkins-plugins-behind-a-corporate-proxy/#comment-29

      Hope that helps.
      Stefan

      Reply
      • Yves
        Yves says:

        Thanks a lot , Stefan.
        This and the comments posted before helped a lot indeed.
        I managed to setup the custom update site (in Jenkins 2.235).
        Here’s a summary of the possible errors, causes and solutions:

        Using “localhost” in the URL gave “Server returned HTTP response code: 503” because we’re not running Jenkins locally.

        We replaced “localhost” by the name of the server where our Jenkins instance runs
        We then received “Server returned HTTP response code: 403” because anonymous users need to be able to read the project’s output file => (you need the plugin “Matrix Authorization Strategy Plugin”)

        Manage Jenkins => Configure Global Security => Project-based Matrix Authorization Strategy: check box “Overall” “Read” for “Anonymous users”
        => the error became “file not found”

        We still needed to grant that permission at project level, i.e. configure the project that creates the file: check box “Job” “Read” for “Anonymous users”
        => the error became “SHA-512 digest mismatch”, showing that the new file is not signed.

        We disabled this by adding JAVA option argument
        -Dhudson.model.DownloadService.noSignatureCheck=true
        => the error became “The update site null does not look like an update center”

        In the pipeline job, we set the name of the output file to “update-center.json” instead of “update-center-updated.json”.
        => No more error !

        I hope this can help others.

        Reply
        • Stefan Maurer
          Stefan Maurer says:

          Hi Yves

          I’m glad my comment was helpful.

          And thanks a lot for sharing further issues (together with the solutions).

          Cheers,
          Stefan

          Reply
        • Gianfranco
          Gianfranco says:

          Hi Yves, I followed your suggestions and fixed the HTTP 403 and SHA-512 digest mismatch problems, but now I have another problem: There were errors checking the update sites: The update site default does not look like an update center. Has this ever happened to you?

          Reply
          • Stefan Maurer
            Stefan Maurer says:

            It looks like the configured update site URL is checked against some patterns. If you store the update file with the name update-center.json instead of update-center-updated.json, downloading the file should work again (as updated in the blog posts code example).

    • Stefan Maurer
      Stefan Maurer says:

      Hi Yves

      Please read my above comment regarding 403.

      Regarding the URLs. Jenkins might have changed that.

      Maybe you could use as well this code in order to avoid an extra redirect:


      pipeline {
      agent any
      stages {
      stage('Download and modify Update-Center Data') {
      steps {
      httpRequest(
      url: "http://updates.jenkins.io/dynamic-stable-" + Jenkins.instance.version + "/update-center.json",
      consoleLogResponseBody: false,
      acceptType: 'APPLICATION_JSON',
      httpProxy: 'http://my.corporate.proxy:8080',
      outputFile: 'update-center.json'
      )
      script {
      updateCenterJson = readFile file: 'update-center.json'
      updateCenterJson = updateCenterJson.replaceAll("http:\\/\\/updates\\.jenkins-ci\\.org\\/download\\/", "http://archives.jenkins-ci.org/")
      }
      writeFile text: updateCenterJson, file: 'update-center-updated.json'
      archiveArtifacts 'update-center-updated.json'
      }
      }
      }
      }

      But please note, that I could verify/test this approach.

      BR
      Stefan

      Reply
  5. keooek
    keooek says:

    Hi Stefan,

    I followed all the suggestions and i have a different error message “There were errors checking the update sites: The update site default does not look like an update center”, i have currently 2.235.1 Jenkins version. I think Gianfranco reported also the same error on 10/09.

    Do you know what the problem could be?

    Thanks in advance.

    Reply
    • Stefan Maurer
      Stefan Maurer says:

      It looks like the configured update site URL is checked against some patterns. If you store the update file with the name update-center.json instead of update-center-updated.json, downloading the file should work again (as updated in the blog posts code example).

      Reply
  6. keooek
    keooek says:

    Hi Stefan,

    Thanks for you help, that was the problem.

    Finally i decided to use external python http server as suggested before, but once executed the pipeline the script kill the http server one hour later (i prefer clean the server with unusual ports opened).

    Additionally i make two differents string modifications, one for war stable version and other for plugins:

    pipeline {
    agent {label ”}
    stages {
    stage(‘Download and modify Update-Center Data’) {
    steps {
    httpRequest(
    url: “https://updates.jenkins.io/update-center.json?id=default&version=” + Jenkins.instance.version,
    consoleLogResponseBody: false,
    acceptType: ‘APPLICATION_JSON’,
    httpProxy: ‘http://’,
    outputFile: ‘update-center.json’
    )
    script {
    updateCenterJson = readFile file: ‘update-center.json’
    updateCenterJson = updateCenterJson.replaceAll(“https:\\/\\/updates\\.jenkins\\.io\\/download\\/war\\/”, “http://archives.jenkins-ci.org/war-stable/”)
    updateCenterJson = updateCenterJson.replaceAll(“https:\\/\\/updates\\.jenkins\\.io\\/download\\/plugins\\/”, “http://archives.jenkins-ci.org/plugins/”)
    }
    writeFile text: updateCenterJson, file: ‘update-center.json’
    archiveArtifacts ‘update-center.json’
    }
    }
    stage(‘Launching web server’) {
    steps {

    sh ”’#!/bin/bash
    export JENKINS_NODE_COOKIE=dontKillMe
    cd /jenkins-slave/workspace/update_center
    nohup sh -c “python -m SimpleHTTPServer 8888” > /dev/null 2>&1 &
    nohup sh -c “sleep 3600 && pkill -f SimpleHTTPServer” > /dev/null 2>&1 &
    echo “Webserver launched”
    ”’

    }
    }
    }
    }

    Reply
  7. Kirill
    Kirill says:

    Hi Stefan,

    Thank you for that solution.
    I followed you and got an error with signature of json-file. How do you solve it?

    Reply
      • Kirill
        Kirill says:

        I think it is not my case. Updater configured to get update-center.json from http://localhost:8080, so no Jenkins’ certificate have to be imported.
        I got error: There were errors checking the update sites: SHA-512 digest mismatch: expected=2a770739f23570e649d0bebed3127968ade082a4100fe6e49373814fb15a6f0755d2e75998ac3b1a248cf62362b832bed6c0896f21c881145b307ed64d4b696d in 'update site 'default''
        It looks like json file is signed or contains checksum.

        I also tried to start java as you recommended and the updater worked fine but I got error abut unreadable data.

        Reply
  8. ribas
    ribas says:

    hello,

    I choose to install same version of jenkins in a zone where I can download. But after installing the package from jenkins webpage I don’t have the corresponding .hpi (and dependance) on my machine. so I can’t retrive the .hpi file to do the installation on the real jenkins server that don’t have internet access

    Reply
  9. Fred
    Fred says:

    I got “SHA-512 digest mismatch” error as Kirill described. This is not SSL cert error. Setting -Dhudson.model.DownloadService.noSignatureCheck=true won’t help. It’s a checksum error on the json file due to change.

    Reply
  10. Denis
    Denis says:

    Thank you for your idea!
    I created python script which downloading update-center.json from jenkins. Rewrite it and save it on my nginx proxy server an updating by crone. Because, I don’t have Anonymous user on my production jenkins.
    All works, but I have error x512. Which certificate I need to add (for what site), If I am using not https, but http url (http://archives.jenkins-ci.org/ or similar mirror) in json file? Certificate of mirror or Jenkins or server where I am hosted my json file?
    Or maybe this error is related that I change my update-center.json
    Thanks in advance.
    I need your advice a lot )

    Reply
    • Stefan Maurer
      Stefan Maurer says:

      Hi Denis

      I’m glad you got some inspiration from this blog post.

      Unfortunately, I don’t fully understand where the described error happens in your setup.

      From what I understood, there are 3 parts:

      1) Download JSON with Python + Cron and store this file locally in ngnix web root
      2) Configure and run ngnix to serve this static JSON file (nginx acts as a simply web server, not really a proxy)
      3) Configure Jenkins to use your ngnix as your update site

      Did I understand correctly and where do you get this 512 error?

      BR
      Stefan

      Reply
  11. Tim
    Tim says:

    FYI you can use a file instead of a url in much the same way: file:///var/jenkins_home/jobs/plugins/workspace/update-center.json

    Reply
    • iostrym
      iostrym says:

      I can’t do this on my jenkins Jenkins 2.332.2. I have an error when providing such a file. are you sure it is working ?

      Reply
  12. Yakup
    Yakup says:

    Hi Stefan,
    first of all awesome post.

    I succeeded with the job and the plugin, i also adjusted the security realm for jenkins in glob config.
    But i’m not able to use localhost as update path.
    For the proxy configuration -> i also excluded localhost in the proxy exceptions.
    My job is a bit different named but this shouldnt be the problem, here is the link:
    http://localhost/job/Rewrite_UpdateCenter_JSON/lastSuccessfulBuild/artifact/update-center.json
    Error message that i receive:
    There were errors checking the update sites: ConnectException: Connection refused (Connection refused)

    Do you have an suggestions or proposals?

    kr
    Yakup

    Reply
  13. iostrym
    iostrym says:

    Hello,

    I can’t succed to provide my own json file. I have 403 error when reaching the .json that I have modified myself (without jenkins, without python but using notepad++) and put in a job jenkins folder (an existing job). anonymous access are granted for jenkins and the corresponding job.
    I have the feeling that the location of the json file inside the job folder is incorrect. maybe I don’t succeed to match the file path and the corresponding URL.

    Here is the file path :
    C:\Users\XXXXXX\.jenkins\jobs\test2\builds\1\artifact\update-center.json
    Here is the URL that I put in update website :
    http://localhost:8080/job/test2/1/artifact/update-center.json

    is my URL correct regarding the path of the json ?

    also because of restriction on my machine I can’t start an apache web service so I have to succed to provide the .json file using the web service of jenkins.

    is it correct ?

    thanks a lot !

    Reply
  14. iostrym
    iostrym says:

    I discover that I had to artifact the file so that it could be reached using a URL but again, same 403 error 🙁

    Reply
  15. Sharath
    Sharath says:

    Hi,

    I’m getting an error, not sure why. Can some one help here
    There were errors checking the update sites: IOException: Unable to tunnel through proxy. Proxy returns “HTTP/1.1 504 Unknown Host”

    Reply

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published.