diff --git a/Linux/jenkins-node/ansible/clean-jenkins-agents.yml b/Linux/jenkins-node/ansible/clean-jenkins-agents.yml new file mode 100644 index 0000000..64da0a7 --- /dev/null +++ b/Linux/jenkins-node/ansible/clean-jenkins-agents.yml @@ -0,0 +1,50 @@ +- name: Playbook to clean jenkins agents by removing workspaces. + hosts: all + + # Tags available: pr, nightly, package, docs + tasks: + - name: Collect workspace sizes + community.docker.docker_container_exec: + container: "{{ agent_name }}" + command: bash -l -c "du -sh * | column -t" + chdir: "/jenkins_workdir/workspace" + register: du_result + become: yes + tags: [always] + + - name: Display workspace sizes + debug: + msg: "{{ du_result.stdout_lines }}" + tags: [always] + + - name: Remove PR directories + community.docker.docker_container_exec: + container: "{{ agent_name }}" + command: bash -l -c "rm -rf pull_requests*" + chdir: "/jenkins_workdir/workspace" + become: yes + tags: [never, pr] + + - name: Remove Nightly directories + community.docker.docker_container_exec: + container: "{{ agent_name }}" + command: bash -l -c "rm -rf *_nightly_deployment*" + chdir: "/jenkins_workdir/workspace" + become: yes + tags: [never, nightly] + + - name: Remove Packages from Branch directories + community.docker.docker_container_exec: + container: "{{ agent_name }}" + command: bash -l -c "rm -rf build_packages_from_branch*" + chdir: "/jenkins_workdir/workspace" + become: yes + tags: [never, package] + + - name: Remove Docs Build directories + community.docker.docker_container_exec: + container: "{{ agent_name }}" + command: bash -l -c "rm -rf build_and_publish_docs*" + chdir: "/jenkins_workdir/workspace" + become: yes + tags: [never, docs] diff --git a/Linux/jenkins-node/ansible/readme.md b/Linux/jenkins-node/ansible/readme.md index 91a55a6..f21336b 100644 --- a/Linux/jenkins-node/ansible/readme.md +++ b/Linux/jenkins-node/ansible/readme.md @@ -7,17 +7,17 @@ New Linux nodes for use on Jenkins are now set up using ansible scripts in this - Ensure you have activated a conda environment with ansible (you may need to use the conda environment set up for use with the ansible-linode repo). - Set up a new node on Jenkins. The easiest way to set up a jenkins node is to copy an existing node: From the jenkins menu, select `New Item`, type in the name for your node, then scroll down to the `copy from` box and enter the node name you wish to copy. - To get the secret for your newly set up node, select your new node from the `Build Executor Status` pane on the left hand side of the jenkins home page. The secret should be displayed as part of the command in the box below `Run from agent command line:`. Note, if you are setting up a node directly on the staging server you will have to use the following command in the jenkins console (`/script`) to obtain the secret: -``` +```java jenkins.model.Jenkins.getInstance().getComputer("").getJnlpMac() ``` - navigate to the [`Ansible folder`](https://github.com/mantidproject/dockerfiles/tree/main/Linux/jenkins-node/ansible) - Update the `inventory.txt` file with the IP address, agent name and agent secret (i.e. Jenkins secret code). Be sure to save this file when update complete! - If creating staging nodes, update the `jenkins-agent-staging.yml` file to specify the correct `jenkins_url`, and `jenkins_identity` variables. To get the `jenkins_identity` use the following command in the jenkins console: -``` +```java hudson.remoting.Base64.encode(org.jenkinsci.main.modules.instance_identity.InstanceIdentity.get().getPublic().getEncoded()) ``` - Run the following command, replacing `staging` with `production` if appropriate and replacing `FedID` with your FedID. -``` +```sh ansible-playbook -i inventory.txt jenkins-agent-staging.yml -u FedID -K ``` - you will be asked for a password - enter your FedID password @@ -30,19 +30,33 @@ When running the `jenkins-agent-staging.yml` or `jenkins-agent-production.yml` p - `agent`: roles tagged with the `agent` tag deploy the docker container that constitutes the jenkins agent. To use a tag, you pass it in to the `ansible-playbook` command with the `-t` flag. For example, if you have already set up the host machine and just want to deploy a jenkins agent: -``` +```sh ansible-playbook -i inventory.txt jenkins-agent-staging.yml -u FedID -K -t agent ``` ## Cleaning nodes -Before cleaning any nodes mark them temporarily offline on Jenkins and ensure no jobs are running on them before cleaning. +- Before cleaning any nodes mark them temporarily offline on Jenkins and ensure no jobs are running on them before cleaning. + +- Update the `inventory.txt` file as above, including only the nodes you intend to clean. + +- The tasks in the cleaning playbook make use of tags to restrict what is cleaned: + + - `pr`: Pull Requests. + - `nightly`: Nightly deployments for main and release next. + - `package`: Build Packages from Branch + - `docs`: Docs build and publish. -The easiest way to clean nodes is using a groovy script on Jenkins. Use the links below for guidance -- [`Remove directories across multiple nodes`](https://developer.mantidproject.org/JenkinsConfiguration.html#remove-directories-across-multiple-nodes) -- [`Remove directories from single node`](https://developer.mantidproject.org/JenkinsConfiguration.html#remove-directories-from-single-node) +- Run the following with the desired tags (which use a comma-separated list): +```sh +ansible-playbook -i inventory.txt clean-jenkins-agents.yml -u FedID -K -t pr,nightly,package,docs +``` + +- Set the nodes you shut down back online. -If this does not work you may need to spin up a new docker container. Use the instructions for Changing docker image below. +### Troubleshooting + +- If this does not work you may need to spin up a new docker container. Use the instructions for Changing docker image below. ## Changing docker image @@ -50,17 +64,17 @@ If you need to update the docker image or spin up a new docker container on a Li - ssh into the node - Stop and remove the container using the following command, replacing machinename with the appropriate name e.g. `isis-cloud-linux-1` -``` +```sh docker stop machinename && docker rm machinename -``` +```sh - Remove any associated volumes using the following command, again replacing machinename with the appropriate name -``` +```sh docker volume rm machinename ``` - for cloud machines, close the ssh connection and follow the instructions for setting up cloud nodes above. - for physical machines navigate to the folder that contains the `deploy.sh` script. This is normally `dockerfiles/jenkins-node/bin`. - Run the following command (you may be able to find it using reverse search) -``` +```sh ./deploy.sh machinename agent_secret "https://builds.mantidproject.org" latest 50G ``` - on success close the ssh connection and check node connects on Jenkins