Skip to main content

Many teams are running Jenkins in their environment. Most of the time it needs to go through a corporate proxy in order to access external resources. It’s relatively easy to configure a proxy in the advanced plugin configuration. But if each domain needs to be white-listed, trouble starts almost certainly.

But let’s start with possible ways of updating Jenkins plugins.

The offline aka manual scenario

If you aren’t allowed to communicate with the internet from within your productive Jenkins environment at all, you still have some options to choose from:

  1. Don’t update
    Not a good idea. Period.
  2. Manually check and download all your plugins with the plugin index https://plugins.jenkins.io/
    Obviously, you don’t want to do this for a typical Jenkins setup with likely more than 70 plugins.
  3. Run a second Jenkins in a zone where you DO have Internet access
    Check for updates and download the desired plugins. Then copy all the .hpi files to your productive Jenkins.

The corporate proxy scenario

If you’re allowed to communicate through a corporate proxy with the Internet, your half-way. But many corporate proxies force you to white-list all required domains or IP’s in order to control access to external resources.

At first, you might think “yeaahhhh… no problem!” and you ask your network security colleagues to enable:

But then, the Jenkins mirroring features hits you.

The meta-data JSON https://updates.jenkins.io/update-center.json provides binary URLs like this: http://updates.jenkins-ci.org/download/plugins/slack/2.10/slack.hpi

But if you call this URL, you get first redirected to http://mirrors.jenkins-ci.org/plugins/slack/2.10/slack.hpi and then another redirect is done depending on your geographic location. In my case, I get redirected to http://ftp-chi.osuosl.org/pub/jenkins/plugins/slack/2.10/slack.hpi

As the status of the mirrors might change, the possibility of returned domains change as well and you’ll find yourself talking to your network security guy quite often.

By-pass mirroring by rewriting meta-data JSON

One possible solution to go round this mirroring feature is to download the update-center.json, rewrite all links and then use the rewritten JSON.

Download and rewrite JSON

Downloading and rewriting the official update-center.json could be done with many technologies. But I choose Jenkins for this as well. Therefore I created a simple Jenkins job name “update-center”, which is scheduled to run once every day.

The following declarative pipeline does the job:

pipeline {
  agent any
  stages {
    stage('Download and modify Update-Center Data') {
      steps {
        httpRequest(
                url: "https://updates.jenkins.io/update-center.json?id=default&version=" + Jenkins.instance.version,
                consoleLogResponseBody: false,
                acceptType: 'APPLICATION_JSON',
                httpProxy: 'http://my.corporate.proxy:8080',
                outputFile: 'update-center-original.json'
        )
        script {
          updateCenterJson = readFile file: 'update-center-original.json'
          updateCenterJson = updateCenterJson.replaceAll("http:\\/\\/updates\\.jenkins-ci\\.org\\/download\\/", "http://archives.jenkins-ci.org/")
        }
        writeFile text: updateCenterJson, file: 'update-center.json'
        archiveArtifacts 'update-center.json'
      }
    }
  } 
}

Some notes regarding the above pipeline:

  • You need to replace my.corporate.proxy:8080 with your actual proxy.
  • We read the current installed Jenkins version with Jenkins.instance.version. This needs to be explicitly approved: https://jenkins.io/doc/book/managing/script-approval/. If this isn’t an option, the version has to be hard-coded.
  • The https://plugins.jenkins.io/http_request plugin is used to download the JSON. You could achieve a similar thing with a simple curl if you don’t want this plugin.
  • You still need to white-list those two domains in your corporate proxy:
    • https://updates.jenkins.io
    • http://archives.jenkins-ci.org
  • Instead of using archives.jenkins-ci.org, you should use a mirror as the official archives server doesn’t provide great performance.

Use the rewritten JSON

Proxy configuration

Go to the advanced Plugin Manager configuration http://localhost:8080/pluginManager/advanced (or Jenkins > Manage Jenkins > Manage Plugins > Advanced) and configure your corporate proxy:

Update Site configuration

You can configure the URL to the JSON at the bottom of the same page.

In my setup, http://localhost/job/update-center/lastSuccessfulBuild/artifact/update-center.json is used.

Now the update will check and download behind your corporate proxy!

Permission

Note: You need to ensure, that anonymous users have read permission to your artifacts. Read more about security configuration here: https://wiki.jenkins-ci.org/display/JENKINS/Standard+Security+Setup

Questions?

Should you have specific questions on this subject, please feel free to comment and share any question you have. If it helped you please like โค๏ธ or share๐Ÿ“Œ this story, so others like you can find it.

Follow N47 on Instagram, Twitter, LinkedIn, Facebook to get updated as soon as possible.

45 Comments

  • Pierre says:

    Hi,

    Very interesting and nice article but in my case (Jenkins 2.150.3) it is not working because of some signature issue. Indeed, everytime I try to check updates, I have an error message on “SHA-512 digest mismatch” that is displayed. I tried to change the java.security file in Jenkins JRE directory but I’m not able to work around this one. Any suggestion ?

    Thanks

    • Stefan Maurer says:

      Hi Pierre

      You might need to import Jenkins certificates like described here: https://docs.oracle.com/cd/E19906-01/820-4916/geygn/index.html

      If you just want to try the setup in general, you could temporary disable SSL errors with ignoreSslErrors: true in the httpRequest plugin as documented here: https://jenkins.io/doc/pipeline/steps/http_request/

      Hope this helps.

      Best regards,
      Stefan

      • Snipe says:

        Hi Stefan,

        I tried to use your approach with the rewriting of the URLs in order to have a fixed “mirror”. However, I ran into the same error as Pierre. It seems it has nothing to do with SSL check. It seems that Jenkins is checking the signature (certificates, correct_digest, correct_digest512) which are generated once the json file is generated. When you modify the json file in any way you would need to re-sign the json file. Is this really working for you? Which version of Jenkins are you using?

    • Varinder says:

      Able to sort the issue using JAVA option argument
      -Dhudson.model.DownloadService.noSignatureCheck=true
      Please read the following
      https://support.cloudbees.com/hc/en-us/articles/115000494608-Why-is-there-Failed-Signature-Check-when-using-update-server-

      • Stefan Maurer says:

        Hi Varinder

        Happy to hear you found a solution. And thanks a lot for sharing it.

        It worked for me as described with Jenkins version 2.160.

        • Snipe says:

          Hi Stefan,

          it seems to work. One error that I needed to fix was that the resulting file must be named “update-center.json”, otherwise it won’t work even with signature check turned off (Message: “There were errors checking the update sites: The update site null does not look like an update center”). However – I am not willing to use this on corporate production server – after all the signature check has a purpose. We decided to open the firewall for the mirrors instead.

  • johny walker says:

    Brilliant solution! Thank you.

  • Switi says:

    I am getting error in my jenkins as Signature verification failed in update site . java.security.cert.CertificateExpiredException:Not After Tue 19 . Unable to install plugins.Any suggestion how to sort out issue?

    • Stefan Maurer says:

      Have you manually imported root certificates in the Trust Store as proposed in my comment from 18/06/2019 at 13:34? They might got expired and need to get imported again.

      • Switi says:

        Thanks. Issue got resolved by updating json file.
        Root cause: Issue was in older json file and proxy also got decommission which using in Jenkins.

        • Stefan Maurer says:

          Glad to hear it’s now working.

          • Switi says:

            Hi Stefan..After upgrading Jenkins version 2.204.1 config file not showing in the managed files. Can you help on this?

          • Stefan Maurer says:

            Hi Switi

            We reached the max depth of comments ๐Ÿ™‚

            Sorry, I don’t get your question. What do you mean by “managed files”?

  • honglus says:

    Following is alternative solution to setup python:SimpleHTTPServer
    It is suitable for docker based Jenkins.
    if you use Kubernetes, put following commands in lifecycle/postStart in container creation.

    cd /var/jenkins_home
    jenkins_version=$(sed -n -e ‘s|\([0-9].*\)|\1|p’ config.xml | sed -e ‘s/ //g’)

    curl -L -o update-center.json -q “https://updates.jenkins.io/update-center.json?id=default&version=$jenkins_version”

    sed -i ‘s|http://updates.jenkins-ci.org/download|http://archives.jenkins-ci.org/|g’ update-center.json
    nohup python -m SimpleHTTPServer 8888 > /dev/null

    then replace https://updates.jenkins.io/update-center.json with ‘http://localhost:8888/update-center.json’ in Jenkins console or baked in docker image

    • honglus says:

      just realized the special char was removed when published and it is security risk to expose /var/jenkins_home. go to /tmp/ instead
      cd /tmp
      jenkins_version=$(sed -n -e ‘s|.version.\([0-9].*\)./version.|\1|p’ config.xml | sed -e ‘s/ //g’)

    • Stefan Maurer says:

      Nice alternative. Thanks honglus.

  • Switi says:

    Hi Stefan..Issue got resolved after install Nodejs plugin.
    I was saying about this managed files .https://plugins.jenkins.io/managed-scripts/

  • Switi says:

    Hi All,
    Am facing issue as no valid crumb was included in the request after upgrade jenkins version.
    Anyone have any idea??

  • Madhu Prakash says:

    Hi Stefan,

    Receiving below error while updating the update center Url.

    There were errors checking the update sites: IOException: Server returned HTTP response code: 403 for URL: http://host:8080/job/Update-Center/lastSuccessfulBuild/artifact/update-center.json?id=default&version=2.222.1

    • Stefan Maurer says:

      Hi Madhu

      Are you sure “host:8080” is pointing to your Jenkins? You could verify by checking access logs (https://wiki.jenkins.io/display/JENKINS/Access+Logging).

      Or maybe you have some custom permissions which forbid access to the used URL.

      Good luck,
      Stefan

      • Gianfranco says:

        Hi Stefan, I’ve the same problem of Madhu.
        Calling http://localhost:8080/…/update-center.json will return HTTP 403, how can I manage this behaviour ?
        Thanks in advance

    • Stefan Maurer says:

      Hi Madhu, Gianfranco and Yves

      Please check the permissions of the job which pulls the update site. You might need to give Read access to anonymous users for that specific job:
      https://wiki.jenkins-ci.org/display/JENKINS/Standard+Security+Setup

      An alternative could be as well the suggestion from honglus to use s simple HTTP Server instead of Jenkins: https://www.north-47.com/knowledge-base/update-jenkins-plugins-behind-a-corporate-proxy/#comment-29

      Hope that helps.
      Stefan

      • Yves says:

        Thanks a lot , Stefan.
        This and the comments posted before helped a lot indeed.
        I managed to setup the custom update site (in Jenkins 2.235).
        Here’s a summary of the possible errors, causes and solutions:

        Using โ€œlocalhostโ€ in the URL gave โ€œServer returned HTTP response code: 503โ€ because weโ€™re not running Jenkins locally.

        We replaced โ€œlocalhostโ€ by the name of the server where our Jenkins instance runs
        We then received โ€œServer returned HTTP response code: 403โ€ because anonymous users need to be able to read the projectโ€™s output file => (you need the plugin โ€œMatrix Authorization Strategy Pluginโ€)

        Manage Jenkins => Configure Global Security => Project-based Matrix Authorization Strategy: check box โ€œOverallโ€ โ€œReadโ€ for โ€œAnonymous usersโ€
        => the error became โ€œfile not foundโ€

        We still needed to grant that permission at project level, i.e. configure the project that creates the file: check box โ€œJobโ€ โ€œReadโ€ for โ€œAnonymous usersโ€
        => the error became โ€œSHA-512 digest mismatchโ€, showing that the new file is not signed.

        We disabled this by adding JAVA option argument
        -Dhudson.model.DownloadService.noSignatureCheck=true
        => the error became โ€œThe update site null does not look like an update centerโ€

        In the pipeline job, we set the name of the output file to โ€œupdate-center.jsonโ€ instead of โ€œupdate-center-updated.jsonโ€.
        => No more error !

        I hope this can help others.

        • Stefan Maurer says:

          Hi Yves

          I’m glad my comment was helpful.

          And thanks a lot for sharing further issues (together with the solutions).

          Cheers,
          Stefan

        • Gianfranco says:

          Hi Yves, I followed your suggestions and fixed the HTTP 403 and SHA-512 digest mismatch problems, but now I have another problem: There were errors checking the update sites: The update site default does not look like an update center. Has this ever happened to you?

          • Stefan Maurer says:

            It looks like the configured update site URL is checked against some patterns. If you store the update file with the name update-center.json instead of update-center-updated.json, downloading the file should work again (as updated in the blog posts code example).

    • Stefan Maurer says:

      Hi Yves

      Please read my above comment regarding 403.

      Regarding the URLs. Jenkins might have changed that.

      Maybe you could use as well this code in order to avoid an extra redirect:


      pipeline {
      agent any
      stages {
      stage('Download and modify Update-Center Data') {
      steps {
      httpRequest(
      url: "http://updates.jenkins.io/dynamic-stable-" + Jenkins.instance.version + "/update-center.json",
      consoleLogResponseBody: false,
      acceptType: 'APPLICATION_JSON',
      httpProxy: 'http://my.corporate.proxy:8080',
      outputFile: 'update-center.json'
      )
      script {
      updateCenterJson = readFile file: 'update-center.json'
      updateCenterJson = updateCenterJson.replaceAll("http:\\/\\/updates\\.jenkins-ci\\.org\\/download\\/", "http://archives.jenkins-ci.org/")
      }
      writeFile text: updateCenterJson, file: 'update-center-updated.json'
      archiveArtifacts 'update-center-updated.json'
      }
      }
      }
      }

      But please note, that I could verify/test this approach.

      BR
      Stefan

  • keooek says:

    Hi Stefan,

    I followed all the suggestions and i have a different error message “There were errors checking the update sites: The update site default does not look like an update center”, i have currently 2.235.1 Jenkins version. I think Gianfranco reported also the same error on 10/09.

    Do you know what the problem could be?

    Thanks in advance.

    • Stefan Maurer says:

      It looks like the configured update site URL is checked against some patterns. If you store the update file with the name update-center.json instead of update-center-updated.json, downloading the file should work again (as updated in the blog posts code example).

  • keooek says:

    Hi Stefan,

    Thanks for you help, that was the problem.

    Finally i decided to use external python http server as suggested before, but once executed the pipeline the script kill the http server one hour later (i prefer clean the server with unusual ports opened).

    Additionally i make two differents string modifications, one for war stable version and other for plugins:

    pipeline {
    agent {label ”}
    stages {
    stage(‘Download and modify Update-Center Data’) {
    steps {
    httpRequest(
    url: “https://updates.jenkins.io/update-center.json?id=default&version=” + Jenkins.instance.version,
    consoleLogResponseBody: false,
    acceptType: ‘APPLICATION_JSON’,
    httpProxy: ‘http://’,
    outputFile: ‘update-center.json’
    )
    script {
    updateCenterJson = readFile file: ‘update-center.json’
    updateCenterJson = updateCenterJson.replaceAll(“https:\\/\\/updates\\.jenkins\\.io\\/download\\/war\\/”, “http://archives.jenkins-ci.org/war-stable/”)
    updateCenterJson = updateCenterJson.replaceAll(“https:\\/\\/updates\\.jenkins\\.io\\/download\\/plugins\\/”, “http://archives.jenkins-ci.org/plugins/”)
    }
    writeFile text: updateCenterJson, file: ‘update-center.json’
    archiveArtifacts ‘update-center.json’
    }
    }
    stage(‘Launching web server’) {
    steps {

    sh ”’#!/bin/bash
    export JENKINS_NODE_COOKIE=dontKillMe
    cd /jenkins-slave/workspace/update_center
    nohup sh -c “python -m SimpleHTTPServer 8888” > /dev/null 2>&1 &
    nohup sh -c “sleep 3600 && pkill -f SimpleHTTPServer” > /dev/null 2>&1 &
    echo “Webserver launched”
    ”’

    }
    }
    }
    }

  • Kirill says:

    Hi Stefan,

    Thank you for that solution.
    I followed you and got an error with signature of json-file. How do you solve it?

    • Stefan Maurer says:

      Hi Kirill

      Have you tried to import the certificates into the trust store as described here: https://www.north-47.com/knowledge-base/update-jenkins-plugins-behind-a-corporate-proxy/#comment-17 ?

      An alternative, but probably just for demo/test purpose, could be to disable the certificate checks:
      -Dhudson.model.DownloadService.noSignatureCheck=true

      Hope that helps.

      • Kirill says:

        I think it is not my case. Updater configured to get update-center.json from http://localhost:8080, so no Jenkins’ certificate have to be imported.
        I got error: There were errors checking the update sites: SHA-512 digest mismatch: expected=2a770739f23570e649d0bebed3127968ade082a4100fe6e49373814fb15a6f0755d2e75998ac3b1a248cf62362b832bed6c0896f21c881145b307ed64d4b696d in 'update site 'default''
        It looks like json file is signed or contains checksum.

        I also tried to start java as you recommended and the updater worked fine but I got error abut unreadable data.

  • ribas says:

    hello,

    I choose to install same version of jenkins in a zone where I can download. But after installing the package from jenkins webpage I don’t have the corresponding .hpi (and dependance) on my machine. so I can’t retrive the .hpi file to do the installation on the real jenkins server that don’t have internet access

  • Fred says:

    I got “SHA-512 digest mismatch” error as Kirill described. This is not SSL cert error. Setting -Dhudson.model.DownloadService.noSignatureCheck=true won’t help. It’s a checksum error on the json file due to change.

  • Denis says:

    Thank you for your idea!
    I created python script which downloading update-center.json from jenkins. Rewrite it and save it on my nginx proxy server an updating by crone. Because, I don’t have Anonymous user on my production jenkins.
    All works, but I have error x512. Which certificate I need to add (for what site), If I am using not https, but http url (http://archives.jenkins-ci.org/ or similar mirror) in json file? Certificate of mirror or Jenkins or server where I am hosted my json file?
    Or maybe this error is related that I change my update-center.json
    Thanks in advance.
    I need your advice a lot )

    • Stefan Maurer says:

      Hi Denis

      I’m glad you got some inspiration from this blog post.

      Unfortunately, I don’t fully understand where the described error happens in your setup.

      From what I understood, there are 3 parts:

      1) Download JSON with Python + Cron and store this file locally in ngnix web root
      2) Configure and run ngnix to serve this static JSON file (nginx acts as a simply web server, not really a proxy)
      3) Configure Jenkins to use your ngnix as your update site

      Did I understand correctly and where do you get this 512 error?

      BR
      Stefan

  • Tim says:

    FYI you can use a file instead of a url in much the same way: file:///var/jenkins_home/jobs/plugins/workspace/update-center.json

    • iostrym says:

      I can’t do this on my jenkins Jenkins 2.332.2. I have an error when providing such a file. are you sure it is working ?

  • Yakup says:

    Hi Stefan,
    first of all awesome post.

    I succeeded with the job and the plugin, i also adjusted the security realm for jenkins in glob config.
    But i’m not able to use localhost as update path.
    For the proxy configuration -> i also excluded localhost in the proxy exceptions.
    My job is a bit different named but this shouldnt be the problem, here is the link:
    http://localhost/job/Rewrite_UpdateCenter_JSON/lastSuccessfulBuild/artifact/update-center.json
    Error message that i receive:
    There were errors checking the update sites: ConnectException: Connection refused (Connection refused)

    Do you have an suggestions or proposals?

    kr
    Yakup

  • iostrym says:

    Hello,

    I can’t succed to provide my own json file. I have 403 error when reaching the .json that I have modified myself (without jenkins, without python but using notepad++) and put in a job jenkins folder (an existing job). anonymous access are granted for jenkins and the corresponding job.
    I have the feeling that the location of the json file inside the job folder is incorrect. maybe I don’t succeed to match the file path and the corresponding URL.

    Here is the file path :
    C:\Users\XXXXXX\.jenkins\jobs\test2\builds\1\artifact\update-center.json
    Here is the URL that I put in update website :
    http://localhost:8080/job/test2/1/artifact/update-center.json

    is my URL correct regarding the path of the json ?

    also because of restriction on my machine I can’t start an apache web service so I have to succed to provide the .json file using the web service of jenkins.

    is it correct ?

    thanks a lot !

  • iostrym says:

    I discover that I had to artifact the file so that it could be reached using a URL but again, same 403 error ๐Ÿ™

  • iostrym says:

    hello,

    here is my error :

    There were errors checking the update sites: IOException: Server returned HTTP response code: 403 for URL: http://localhost:8080/job/update-center/lastSuccessfulBuild/artifact/update-center

    anonymous have all access and the URL works in my browser. even when I’m not logged to jenkins.

  • Sharath says:

    Hi,

    I’m getting an error, not sure why. Can some one help here
    There were errors checking the update sites: IOException: Unable to tunnel through proxy. Proxy returns “HTTP/1.1 504 Unknown Host”