Setting up a local CI server

[gnl42]

How to set up a local CI server for the Mylyn tests.

Starting with the notes by @BeckerFrank https://github.com/eclipse-mylyn/org.eclipse.mylyn/tree/main/org.eclipse.mylyn.releng/multipass

I'm getting this error:
fatal: [mylynmstr01]: FAILED! => {"changed": false, "cmd": ["/usr/bin/python3", "-m", "pip.__main__", "install", "docker"], "msg": "\n:stderr: error: externally-managed-environment\n\n× This environment is externally managed\n╰─> To install Python packages system-wide, try apt install\n python3-xyz, where xyz is the package you are trying to\n install.\n \n If you wish to install a non-Debian-packaged Python package,\n create a virtual environment using python3 -m venv path/to/venv.\n Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make\n sure you have python3-full installed.\n \n If you wish to install a non-Debian packaged Python application,\n it may be easiest to use pipx install xyz, which will manage a\n virtual environment for you. Make sure you have pipx installed.\n \n See /usr/share/doc/python3.12/README.venv for more information.\n\nnote: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.\nhint: See PEP 668 for the detailed specification.\n"}

The pip module 'docker' is installed so...

[BeckerFrank]

How to set up a local CI server for the Mylyn tests.

Starting with the notes by @BeckerFrank https://github.com/eclipse-mylyn/org.eclipse.mylyn/tree/main/org.eclipse.mylyn.releng/multipass

I'm getting this error: fatal: [mylynmstr01]: FAILED! => {"changed": false, "cmd": ["/usr/bin/python3", "-m", "pip.__main__", "install", "docker"], "msg": "\n:stderr: error: externally-managed-environment\n\n× This environment is externally managed\n╰─> To install Python packages system-wide, try apt install\n python3-xyz, where xyz is the package you are trying to\n install.\n \n If you wish to install a non-Debian-packaged Python package,\n create a virtual environment using python3 -m venv path/to/venv.\n Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make\n sure you have python3-full installed.\n \n If you wish to install a non-Debian packaged Python application,\n it may be easiest to use pipx install xyz, which will manage a\n virtual environment for you. Make sure you have pipx installed.\n \n See /usr/share/doc/python3.12/README.venv for more information.\n\nnote: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.\nhint: See PEP 668 for the detailed specification.\n"}

The pip module 'docker' is installed so...

I think that is fixed with commit 0c4c973

[gnl42]

Looking better. Now I get:

failed: [mylynadm01] (item=/cacerts/root_ca) => {"ansible_loop_var": "item", "changed": false, "cmd": "/usr/bin/rsync --delay-updates -F --compress --archive --rsh='/usr/bin/ssh -S none -i first_setup/ubuntu_key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null' --out-format='<<CHANGED>>%i %n%L' [email protected]:/cacerts/root_ca /home/certs/", "item": "/cacerts/root_ca", "msg": "Warning: Permanently added '10.72.34.118' (ED25519) to the list of known hosts.\r\nrsync: [Receiver] mkdir \"/home/certs\" failed: Permission denied (13)\nrsync error: error in file IO (code 11) at main.c(791) [Receiver=3.2.7]\n", "rc": 11}

[gnl42]

Manually created the folder. Gets me (ubuntu windows subsystem):

fatal: [mylynmstr01]: FAILED! => {"changed": false, "msg": "Error pulling image gitlab/gitlab-ee:16.11.1-ce.0 - 404 Client Error for http+docker://localhost/v1.45/images/create?tag=16.11.1-ce.0&fromImage=gitlab%2Fgitlab-ee: Not Found (\"manifest for gitlab/gitlab-ee:16.11.1-ce.0 not found: manifest unknown: manifest unknown\")"}

[BeckerFrank]

#537 should fix this problem

[gnl42]

Switching to my mac:
fatal: [mylynmstr01]: FAILED! => {"changed": false, "msg": "Error building 127.0.0.1:5000/jenkins-2_375_2 - code: 1, message: The command '/bin/sh -c jenkins-plugin-cli -f /usr/share/jenkins/ref/plugins.txt' returned a non-zero code: 1, logs: ['Step 1/9 : ARG JENKINS_IMAGE_VERSION=jenkins/jenkins:latest-jdk11', '', 'Step 2/9 : FROM $JENKINS_IMAGE_VERSION', '', 'Pulling from jenkins/jenkins', ... ' ---> Running in 9611d801e549', '\\x1b[91mMultiple plugin prerequisites not met:', '', 'plugin-util-api (3.2.1) requires a greater version of Jenkins (2.387.1) than 2.375.2,', '', 'font-awesome-api (6.3.0-2) requires a greater version of Jenkins (2.387.1) than 2.375.2,', '', 'checks-api (2.0.0) requires a greater version of Jenkins (2.387.1) than 2.375.2,', '', 'jquery3-api (3.7.0-1) requires a greater version of Jenkins (2.387.1) than 2.375.2,', '', 'credentials (1311.vcf0a_900b_37c2) requires a greater version of Jenkins (2.387.1) than 2.375.2,', '', 'matrix-project (818.v7eb_e657db_924) requires a greater version of Jenkins (2.375.4) than 2.375.2,', '', 'bootstrap5-api (5.2.3-1) requires a greater version of Jenkins (2.387.1) than 2.375.2,', '', 'junit (1217.v4297208a_a_b_ce) requires a greater version of Jenkins (2.387.1) than 2.375.2,', '', 'echarts-api (5.4.0-4) requires a greater version of Jenkins (2.387.1) than 2.375.2', '\\x1b[0m', ' ---> Removed intermediate container 9611d801e549']"}

[gnl42]

Updated jenkins version to 2.375.4 and the setup seems to complete successfully on my mac.

Now, what do I do with the setup? How do I use it?

What are the three VMs for: docker, swarm, single. The "or" seems to imply I really only need one

[BeckerFrank]

for an setup I actual use

multipass % ansible-playbook create_vm.yml --ask-become-pass --extra-vars="docker_run_mode=all"
ansible-playbook mylyn_setup.yml --extra-vars="docker_run_mode=container" --extra-vars="remote_domain=mylyn.frank-becker.de" --skip-tags artifactory,bugzilla,jenkins,gitlab

from the last command I only remove one element from skip-tags to setup an Instance with the removed Item.

Then I use an eclipse launcher and start the tests. By default the org.eclipse.mylyn.commons.sdk.util.TestConfiguration first checks if we have a service for localhost (default https://mylyn.local). If a service of the requested kind is found in .../mylyn_idx/service this is used.

[gnl42]

Can I use an IP address for the remote_domain?

[BeckerFrank]

No I do not think that an IP address can work because we use traefik as a proxy.

[gnl42]

Getting there.

@BeckerFrank How do I reset everything ansible has done to my machine so I can start from a clean slate again?

[BeckerFrank]

I use the command documented in Releng Multipass Readme.md 6.

For my Mac this are the following 4 commands

multipass delete --all                                                                                                                             
multipass purge
sudo vi /var/db/dhcpd_leases
vi ~/.ssh/known_hosts

in /var/db/dhcpd_leases I delete 21 lines for 3 instances and
in ~/.ssh/known_hosts I delete 9 lines for 3 IP Adresses

[gnl42]

Wasn't sure it that cleaned up everything.

So I picked up a pi5 and installed ubuntu 24.04 on it. With a couple of tweaks have it "mostly" running. I ran:

1. ansible-playbook create_vm.yml --ask-become-pass --extra-vars="docker_run_mode=container"
2. ansible-playbook mylyn_setup.yml --extra-vars="docker_run_mode=container"
3. ansible-playbook mylyn_setup.yml --skip-tags base_software --extra-vars="docker_run_mode=container"
4. multipass delete --all
5. multipass purge
6. vi ~/.ssh/known_hosts (/var/db/dhcpd_leases doesn't exist)
7. ansible-playbook create_vm.yml --ask-become-pass --extra-vars="docker_run_mode=all"
8. Add mylyn.local (and subdomains to local dns server)
9. ansible-playbook mylyn_setup.yml --extra-vars="docker_run_mode=container" --extra-vars="remote_domain=mylyn.local"
   • Getting an error about not being able to resolve gitlab.mylyn.local. What does your DNS setup look like?

So a bunch of question:

• What are the three VMs for:

  Name                    State             IPv4             Image
  mylynadm01              Running           10.194.76.146    Ubuntu 24.04 LTS
  mylynmstr01             Running           10.194.76.174    Ubuntu 24.04 LTS
                                            172.18.0.1
                                            172.17.0.1
  mylynwrk01              Running           10.194.76.168    Ubuntu 24.04 LTS
                                            172.17.0.1```
  

• I can access mylyn_idx locally (https://10.194.76.174:8001/mylyn_idx) but there is nothing listing on 80/443 that forwards requests to the VMs
• The links on the services page give me 404 ?
• I find I have to run create_vm.yml twice. The first time the final step fails because ssh is unable to connect to the VMs. The second run works. I tried adding a delay before that step, but it didn't help. I don't know if the delay (20 seconds) was enough or if there is something else going on.
• I'm seeing this complaint:

  Tuesday 04 June 2024  23:06:56 -0700 (0:00:02.176)       0:03:02.967 **********
  Tuesday 04 June 2024  23:06:56 -0700 (0:00:02.176)       0:03:02.966 **********
  [DEPRECATION WARNING]: The connection's stdin object is deprecated. Call display.prompt_until(msg) instead. This feature will be removed in version 2.19. Deprecation warnings can be disabled by
  setting deprecation_warnings=False in ansible.cfg.```
  No idea how to get rid of it.
  

• Why the delete/purge and then run the commands "again"?
• I'm adding my observations/tribulations to the README.MD. Should I removed the "swarm" and "single" versions?

[BeckerFrank]

Here the list of commands with a short description that I ran on my Mac.

1. if I had already a running Multipass Environment the I execute the following otherwise go to 2.

   • multipass delete --all
     Delete instances and snapshots
   • multipass purge
     Purge all deleted instances permanently
   • sudo vi /var/db/dhcpd_leases
     This file has a mapping for multipass vm to IP Adresses (7 lines per instance)
     Example:

     {
       name=mylynmstr01
       ip_address=192.168.64.3
       hw_address=1,52:54:0:c0:9c:55
       identifier=1,52:54:0:c0:9c:55
       lease=0x6661c93d
     }
     ``` and I delete all lines
     

   • vi ~/.ssh/known_hosts
     For all previous existing Multipass IP Addresses I delete the entries (for me 3 lines per instance)

2. multipass % ansible-playbook create_vm.yml --ask-become-pass --extra-vars="docker_run_mode=all"
   This ask for your password a the with docker_run_mode=all I tell to create 3 new instances.
   Example:

   multipass list
   Name                    State             IPv4             Image
   mylynadm01              Running           192.168.64.2     Ubuntu 24.04 LTS
   mylynmstr01             Running           192.168.64.3     Ubuntu 24.04 LTS
                                             172.17.0.1
   mylynwrk01              Running           192.168.64.4     Ubuntu 24.04 LTS
                                             172.17.0.1
   

3. ansible-playbook mylyn_setup.yml --extra-vars="docker_run_mode=container" --extra-vars="remote_domain=mylyn.frank-becker.de" --skip-tags artifactory,jenkins,gitlab
   This Ansible Playbook puts the requested Services into the multipass instances.
   With --skip-tags you can define which service should not go into the instances. In this Example we do not want an Artifactory Jenkins and Gitlab service.

4. Verify the setup with your Browser an Address https://traefik.mylyn.local
   You should see the following.
   

5. Show the defined Services with https://mylyn.local/mylyn_idx/
   

6. defined Services as json with https://mylyn.local/mylyn_idx/service
   

The docker_run_mode controls which service is deployed in a multipass vm.

1. all (only valid in ansible-playbook mylyn_setup.yml) creates 3 VM's
2. container
   use 2 VM's
   • VM for certificate
   • VM for all docker containers
3. swarm
   use 3 VM's
   • VM for certificate
   • VM for docker swarm manager
   • VM for docker swarm worker
4. single
   use 1 VM's for certificate and all docker containers

Actual I prefer all for mylyn_setup.yml and Container for mylyn_setup.ym so you maybe find some sort correctly workin bits in the other modes

[gnl42]

Still can't get it to work on my mac.

First time I run ansible-playbook create_vm.yml --ask-become-pass --extra-vars="docker_run_mode=all" it fails at the end with the ssh command failing to connect to the VM.
If I run it again it runs to completion, no idea if a second run messes anything up.

Once I have a clean run none of the .mylyn.local hosts are defined, so https://traefik.mylyn.local/ fails to resolve traefik.mylyn.local

What is supposed to define and resolve the mylyn.local domains?

[BeckerFrank]

We have default_etc_hosts in the inventory.yml so maybe something with the setup of file /etc/hosts (if on linux or macOS) is not as expected.

Her what is into the file /etc/hosts in my Mac environment

192.168.64.2 mylynadm01
192.168.64.3 mylyn-local traefik.mylyn.local mylyn-local.mylyn.local mylyn.local jenkins.mylyn.local hello.mylyn.local artifactory.mylyn.local gitlab.mylyn.local
192.168.64.4 mylynwrk01 mylynwrk01.mylyn.local

[gnl42]

I've only seen /etc/hosts modified once (and it was a mess), I'll see if I can track that one down.

Modding hosts helps.

What is supposed to be running on mylyn.frank-becker.de that allows you to access the mylyn.local domains?

[BeckerFrank]

mylyn.frank-becker.de was a test to see if I could use my local setup as a replacement for the now discontinued mylyn.org test instances.

You need to activate line # additional_launch_parms: --network en0 # acces from external and open a port forwarding in your firewall.

[gnl42]

Not seeing anything waiting on ports 80/443.

What process should be listening on the ports?

[gnl42]

@BeckerFrank I can't figure out what is going on anymore.
Here is what I'm doing and what I'm getting:

1. multipass delete --all && multipass purge; sudo cp /dev/null /var/db/dhcpd_leases && cp /dev/null ~/.ssh/known_hosts && rm -fr first_setup && git checkout inventory.yml
2. /etc/hosts:
   192.168.64.2 mylynadm01
192.168.64.3 mylyn-local traefik.mylyn.local mylyn-local.mylyn.local mylyn.local jenkins.mylyn.local hello.mylyn.local artifactory.mylyn.local gitlab.mylyn.local
192.168.64.4 mylynwrk01 mylynwrk01.mylyn.local
3. ansible-playbook -vv create_vm.yml --ask-become-pass --extra-vars="docker_run_mode=container" | tee step1.log
4. TASK [multipass_create_vm_setup : add etc hosts if not included mylynadm01] **** fails with:
   ssh: connect to host 192.168.64.2 port 22: Host is down
5. /etc/hosts now:
   192.168.64.16 mylynadm01
192.168.64.17 mylyn-local traefik.mylyn.local mylyn-local.mylyn.local mylyn.local jenkins.mylyn.local hello.mylyn.local artifactory.mylyn.local gitlab.mylyn.local
192.168.64.4 mylynwrk01 mylynwrk01.mylyn.local
6. multipass list:
   Name State IPv4 Image
mylynadm01 Running 192.168.64.16 Ubuntu 24.04 LTS
mylynmstr01 Running 192.168.64.17 Ubuntu 24.04 LTS
7. /var/db/dhcpd_leases contains 17 entries: 9 for mylynmstr01, 7 for mylynadm01, and 1 for mylynwrk01.
   The ip addresses go from 192.168.64.2 to 192.168.64.18

[BeckerFrank]

When you have an existing environment and want to start from scratch you not only have to execute
multipass delete --all && multipass purge;
to destroy the actual environment .

You need to cleanup some other files. For me on a Mac this are

• /var/db/dhcpd_leases
  where we have the mapping between mac and IP (7 lines for each instance). If I not remove the existing lines I end up with changed IP entries for the mutlipass vm.

  {
        name=mylynwrk01
        ip_address=192.168.64.4
        hw_address=1,52:54:0:8:9:17
        identifier=1,52:54:0:8:9:17
        lease=0x667af1fe
  }
  {
        name=mylynmstr01
        ip_address=192.168.64.3
        hw_address=1,52:54:0:f5:12:63
        identifier=1,52:54:0:f5:12:63
        lease=0x667af1e5
  }
  {
        name=mylynadm01
        ip_address=192.168.64.2
        hw_address=1,52:54:0:24:cd:41
        identifier=1,52:54:0:24:cd:41
        lease=0x667af1cf
  }
  

• ~/.ssh/known_hosts
  here we the old values for the multipass vm that we need to remove so we use the new values when we create the new multipass vm instances. For me this are 3 lines per IP Adress

  192.168.64.2 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBrbukuho6yzg7R9g3wBb+hYtRM0S3iHVIIw45/laggr
  192.168.64.2 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCfykLV0NgqzNKtU6Y8IYK3all2RIMTDQL+raXNeO/swcyZBCEsg0U24XYRJdn3PXuTJFYiMIoDMy2k5Mi+7ABlUjeBxPt1+YHxANmor9LU3F6ccii9lAl5Gu8rYTbXMa6qHDSNPGAl9AEuTZngZLeX/QJSZvKxUG9/n8coGqKKO60Jb/JhzKNayl4mtmumDdWkOVgc+YsHVZh33qTsJF9gkgtMLyNYz3JOIMctVA8ct3ZcpUP2KbU/AqCaIEOkonFG5NrsJXwZ+MRWWTpZrUGQ7EXIiflML163GQfulC0CjNjGxnX/EUvuCbV+JcromZsCSSOGp1DlK0w9bp/pDedN4oILzaZ7ZY8t64dnUGAtf0TWTm8kX7UEqVh7b/CxlDRF1ZStag7gRgaPitk2ocKOsCbsk79r2Z3euivkhu5XCohlY5l/wXBz92wgANGnc8jQayU/rkfzLvfzzFMoxNxUCwqNjUEcnYaePfU1OLg6JevC6MzdGIDvzcy7/C751s=
  192.168.64.2 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH5mcReTe/ZYcpZCI0kzY7/YpoX3P+2iRDTrNik0JWtclghPMucuJE9pX8ymSuvklSesTKFvopf1JBYhSoHlNfU=
  192.168.64.3 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFkuKMoVDT4egqXcTTG9xrligUMbTA5DZQIVc323Y6nm
  192.168.64.3 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4Cz70B/XoKgklhMntY2Gu2thqugNR2B0SR7RAucPUrxpPDETWooXFobDZTVricdOnhtChqCa1SG4C4+xIUw/PhDZJWi+Fc+hDNNe292qu01Ri7O054Kais2JhbbNwIXoH7kDncUyI38WXsSHWG4IMH0aU+htNM8JESd2vZvm3bQWbTp4cWVWlWKZRai8rpkHM7VmKSaDS30dzsBu4QyGe7WwhmxMgDarOBlCfJeatrdYEiuayhgJOv/hepN4H1a7jS4Hai+nFIWhj0onX+8YB3R2lNNHmXRqlHQ10zGFYQrPCrGyWEIS62IZmEHFd5BI4ol3SQkTroisjV1fEMx5tFyJD3HJjnbveUJMgW5AMTYieonldTfbo7HBj6w+AHilM7RA54wuysPAT19f+qVD9J8C5k93Le4efNiC/BSnAB6kExVeWpE0OTwOfpdaKnKgInGInZru3MaeAwomMAeeDiz3U4dLWNweouVK0j8MPXU6Q/b5i7DQuucQGDoNl0O0=
  192.168.64.3 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO4wpKSfDlPmc6q55UVN7lnUY1+ESb9c1eFswNSRCsOYe3KjCoQaCQWt4XA0DG2dx7frJlx5UIBYsfsAO3gIL70=
  192.168.64.4 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFsoY9mwAVkssl0K2BNrGYzPyLxBvTyQYvBvu53qLh1D
  192.168.64.4 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCg9q13fMaHL0cVQaAOq98QPaADSIwWPtPYj69+Ql2uaVNJtD2uvzUNRFIrZJ8FAuVQEOGaZkyvUWEqPQl909Vr3rXu8fDng0PkUMHv+gR5owgMFPYi6jf4UeKyhPsTPMXgzmP9J+e7hFQFQVw/Hsws1U1Yuzn++k9GdDU4IQV23mwWYATwGcI1ZIHJe4amSXCUrAJnM9IZSBS2Vdexg+E163gQdOWg+psH5fgfjGfJS12xpdJJjwhFGcxhyxBfLrNtZkHueFmcdfwBKPpjcaSc2vi8bv/cf5gkl8a2GkK7/HAZglnojy3x2AItFrMQbjxSUR1nAk1kP7H1fCUUGtLeWb+W0e1tkuTSBf3Ky7KwlijpW+zyrRLFA1ryL31dg/iPhEBRoEb/J80nhfyO2DymuPmBZWw+mZLHIr1yiNxsnhoJIZt5PXJ1RYQEvK68hBVvKxabxlIe3C2PZxf37cLDuAUfbBf9iWqY4LyNSSDBkSNcLqfGf9vehgZ6bbkIfv8=
  192.168.64.4 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAafRCSHQ0rGRiAJG6CE2hBl+qqINUwZvcNMSBjrGZxLj9QRb8mUX7BQe52wVT7aF9AhWZfO3k29WxVHKZ/7ywo=
  

Maybe it is the best if you can do the following.

• destroy all multipass instances
• delete all lines /var/db/dhcpd_leases
• remove all lines of previous used IP Adresses from /etc/hosts
• reboot your system
• use docker_run_mode=all in create_vm.yml (the other values are no longer used and tested.

This way multipass should start with 3 multipass instance 192.168.64.2 - 192.168.64.4

[gnl42]

So I discovered that all the errant IP addresses were due to still running qemu/multipassd processes.
Once I rebooted to get rid of the processes everything built properly.

all .mylyn.local domains work, but I'm still not seeing anything listening on ports 80/443

[BeckerFrank]

What did you see when you enter the URL https://traefik.mylyn.local/dashboard?

[gnl42]

[BeckerFrank]

Please test if you get a response from https://mylyn.local/mylyn_idx/ and https://mylyn.local/mylyn_idx/service

The first url should return some Header Information and the a list of defined services. The last will return a json with all the services

[gnl42]

They all work fine.

There is just no process listing in on port 80 or 443:

George@mylynmac multipass % telnet localhost 80
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
Trying ::1...
telnet: connect to address ::1: Connection refused
telnet: Unable to connect to remote host
George@mylynmac multipass % telnet localhost 443
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
Trying ::1...
telnet: connect to address ::1: Connection refused
telnet: Unable to connect to remote host

[BeckerFrank]

I think the process you are looking for is inside of the docker container traffik.
What do you see if you enter the command docker ps
I get

ubuntu@mylynmstr01:~$ docker ps
CONTAINER ID   IMAGE                                               COMMAND                  CREATED       STATUS                 PORTS                                                           NAMES
5f43f658b868   127.0.0.1:5000/bugzilla:4.4                         "/opt/start.sh"          12 days ago   Up 12 days             0.0.0.0:8090->80/tcp                                            bugzilla-4_4
cd44e0c85305   127.0.0.1:5000/jenkins-latest-jdk11:1               "/usr/bin/tini -- /u…"   12 days ago   Up 12 days             0.0.0.0:8083->8080/tcp, 0.0.0.0:50003->50000/tcp                jenkins-latest-jdk11
63df6f4f59bd   127.0.0.1:5000/mylyn-gitlab-ce                      "/assets/wrapper"        12 days ago   Up 12 days (healthy)   80/tcp, 443/tcp, 0.0.0.0:8888->8888/tcp, 0.0.0.0:2222->22/tcp   gitlab
dece9997c849   docker.bintray.io/jfrog/artifactory-cpp-ce:latest   "/entrypoint-artifac…"   12 days ago   Up 12 days             0.0.0.0:8081-8082->8081-8082/tcp                                artifactory
45daccc5c45a   127.0.0.1:5000/mylyn_idx:1                          "python3 app.py 8001"    12 days ago   Up 12 days             0.0.0.0:8001->8001/tcp                                          mylyn_idx
d4dc7e830da4   tecnativa/docker-socket-proxy:latest                "docker-entrypoint.s…"   12 days ago   Up 12 days             0.0.0.0:2375->2375/tcp                                          docker-socket-proxy
df61a3a55260   registry:2                                          "/entrypoint.sh /etc…"   12 days ago   Up 12 days             0.0.0.0:5000->5000/tcp                                          registry
b1a8ef73707c   traefik:latest                                      "/entrypoint.sh trae…"   12 days ago   Up 12 days             0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                        traefik

[gnl42]

I realize all that.
I want to do the same my setup.
I have a local dns server set up for a local domain

George@mylynmac ~ % nslookup mylynmac.glindholm.me
Server:         192.168.0.235
Address:        192.168.0.235#53

Name:   mylynmac.glindholm.me
Address: 192.168.0.151

with the idea that I can use a second machine for the services needed by the unit tests
I run ansible with my domain:
ansible-playbook mylyn_setup.yml -vv --extra-vars="docker_run_mode=container" --extra-vars="remote_domain=mylynmac.glindholm.me" | tee step2.log
and I've uncommented the addtional_launch_parameters to use my ethernet interface:

George@mylynmac multipass % grep  additional_launch_parms inventory.yml
    additional_launch_parms: ''
        additional_launch_parms: --network en1 # acces from external

What's missing?

[BeckerFrank]

Did you try to access https://mylynmac.glindholm.me/mylyn_idx/ from outside of your network. This url is not working when you are in your local network. I never got a response from https://mylyn.frank-becker.de/mylyn_idx/ but when I ask others the can access the page.

So if you have setup dyndns to use mylynmac.glindholm.me and forward port 80 and 443 to your local server and the server is active the I should see the list of services with https://mylynmac.glindholm.me/mylyn_idx/ .

[gnl42]

I haven't tried from outside my network, kind defeats the purpose.

So what is the purpose of --network en1 then?
I assumed it was to tell traefik (or some other process) to listen on the interface

[BeckerFrank]

Yes, you only need the --network if you want to share your test setup with others.

With --network en0 multipass is connected to the host network, so that port 80 and 443 requests to the IP address of the host can be processed within the multipass vm and thus forwarded to the docker traefik container. This only works if no other process on the host listen to the ports 80 and 443.

I add the --network to test if it is possible to share an test server so that we can use an fix url like https://mylyn.frank-becker.de to run Jenkins Pipeline test.

I currently have a Mac mini server from 2011, once I have replaced it with the next version of a Mac mini (maybe M4) I can share this setup with others and we can activate the integration test again.

[gnl42]

That's the thing though, there is no process listening on ports 80/443.

If you do sudo lsof -i tcp:80 on your box, what do you see?

[BeckerFrank]

when I do a sudo lsof -i tcp:80or sudo lsof -i tcp:443on my MacBook I do not get any output.
When I'm inside the mylynmstr01 ubuntu (multipass shell mylynmstr01)
I get the following output.

sudo lsof -i tcp:80
COMMAND     PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
docker-pr 21811 root    4u  IPv4  63765      0t0  TCP *:http (LISTEN)

docker exec -it traefik lsof -i tcp:80
1	/usr/local/bin/traefik	0	/dev/null
1	/usr/local/bin/traefik	1	pipe:[63282]
1	/usr/local/bin/traefik	2	pipe:[63283]
1	/usr/local/bin/traefik	3	socket:[64535]
1	/usr/local/bin/traefik	4	anon_inode:[eventpoll]
1	/usr/local/bin/traefik	5	pipe:[63834]
1	/usr/local/bin/traefik	6	pipe:[63834]
1	/usr/local/bin/traefik	7	socket:[64538]
1	/usr/local/bin/traefik	8	socket:[64539]
1	/usr/local/bin/traefik	9	anon_inode:inotify
1	/usr/local/bin/traefik	10	socket:[62339]
1	/usr/local/bin/traefik	11	socket:[65280]

[gnl42]

Sigh, thanks.

I was hoping to be able to access the multipass instances from my development machine, but I guess that will have to wait.
Still trying to figure this out, need to read up more on bridged networks.

What would it take for me to create a straight docker setup from the multipass setup?

[BeckerFrank]

You can access the multipass instances from your network with

• https://mylyn.local/mylyn_idx/
  you get a list of defines services
• https://mylyn.local/mylyn_idx/service
  same as with but result is in json
• https://mylyn.local/bugzilla-4_4/
  for Bugzilla 4.4
• https://traefik.mylyn.local/dashboard/#/

If you want to share your environment you can use remote_domain but this is optional.

[gnl42]

That's the problem, I can't connect to any of the services from outside the box.

*.mylyn.local is only defined for the box so any of services is not exposed to the network, and remote_domain does not expose ports 80/443 to my network.

From what I can gather, even though I specify remote_domain and use --network en0, the network bridge set up by multipass does not expose the ports.

As an experiment I created a ubutu vm and ran an EchoServer there.

multipass launch mantic --network en0 --name echo
George@mylynmac ~ % multipass list
Name                    State             IPv4             Image
echo                    Running           192.168.64.7     Ubuntu 23.10
                                          192.168.0.166

at least the instance is on my local network, but using an IP address that is local to the box, so I could still not connect to it.
I stopped the instance and run qemu manually with the extra arguments -netdev socket,id=vmnic,listen=192.168.0.151:9090 and now I could connection to port 9090 from the outside. I have no idea what was listening but something accetped the connection request.

I also tried adding the values to multipassd-vm-instances.json but they are wiped out by something when I started vm.

[BeckerFrank]

What is the content of /etc/hosts in your Host ( not multipass vm).
For me with my Mac I have

192.168.64.2 mylynadm01
192.168.64.3 mylyn-local traefik.mylyn.local mylyn-local.mylyn.local mylyn.local jenkins.mylyn.local hello.mylyn.local artifactory.mylyn.local gitlab.mylyn.local
192.168.64.4 mylynwrk01 mylynwrk01.mylyn.local

and can access mylyn.local from my Mac.

[gnl42]

I have the same

[BeckerFrank]

What happens if you use the branch https://github.com/eclipse-mylyn/org.eclipse.mylyn/tree/CIServerProperty and run local-All-BugzillaTests?
For my this runs 150 Test with 12 Errors and 4 Failures

Using the Webbrowser Safari with url https://mylyn.local/mylyn_idx/ gives my a Dialogbox with "not private". Details shows a Warning about the Certificate but I can continue visiting the Web Site and get the list of Services.

and then

Where do you have Errors?

[gnl42]

*mylyn.local seems to work fine, https://mylynmac.glindholm.me does not

P.S. Took me two hours to figure out how to get around:
/Users/George/.swt/lib/macosx/aarch64/libswt-cocoa-4964r8.jnilib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e' or 'arm64'))

[BeckerFrank]

*mylyn.local seems to work fine, https://mylynmac.glindholm.me does not

If you want to share your environment I can try to test if the setup is assessable.
Is https://mylynmac.glindholm.me/mylyn_idx/ configuered that I should try it?

[gnl42]

I wish you could, but it's on a local network with a local DNS server

George@mylynmac multipass % nslookup mylynmac.glindholm.me
Server:         192.168.0.235
Address:        192.168.0.235#53

Name:   mylynmac.glindholm.me
Address: 192.168.0.151

George@mylynmac multipass % telnet mylyn.local 80
Trying 192.168.64.3...
Connected to mylyn-local.
Escape character is '^]'.
^]
telnet> Connection closed.
George@mylynmac multipass % telnet mylynmac.glindholm.me 80
Trying 192.168.0.151...
telnet: connect to address 192.168.0.151: Connection refused
telnet: Unable to connect to remote host

What is supposed to sit between en0 and bridge100?

[BeckerFrank]

For access from a local network, you do not need mylynmac.glindholm.me. This is done with mylyn.local.

I extend the setup so that it is also accessible to others who are not in the local network (replacement for http://mylyn.org/).

This requires a dyndns setup.

• For me, this meant creating a subdomain mylyn.frank-becker.de and configuring my router so that it is accessible for this subdomain.

• The next step was to set up port forwarding for ports 80 and 443 to my computers with the multipass VM on them.

• Next I need to make sure that the requests for port 80 and 443 are forwarded to multipass. This is done with additional_launch_parms: --network en0 in inventory.yml.

• The last step was to tell traefic that it should not only respond to mylyn.local. This is achieved with additional labels for the subdomain.

Now someone from outside you network can access the services using the subdomain url

[gnl42]

Something is missing, but what?

• Using dnsmasq for dns server
• local network so no need to configure router for ports 80/443
• using additional_launch_parms: --network en0 in inventory.yml.
• Is the traefik additional labels configuration part of the scripts or something done manually? That seems to be the missing step

tcpdump shows connection request coming in:

7:22:53.850405 IP 192.168.0.120.50408 > mylynmac.glindholm.me.https: Flags [S], seq 1583759449, win 64240, options [mss 1460,sackOK,TS val 1680518223 ecr 0,nop,wscale 7], length 0

mylynmstr01 shows process listening on port 443:

ubuntu@mylynmstr01:~$ sudo lsof -i tcp:443
COMMAND     PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
docker-pr 21331 root    4u  IPv4  61962      0t0  TCP *:https (LISTEN)

Best I can tell en0 is not passing the packets to bridge100.

And why is bugzilla using the external hostname in both our instances:

[BeckerFrank]

No nothing is missing.
The url https://mylyn.local/bugzilla-4_4/ is used but in the Properties and two Bugzilla preferences we use the remote_domain_name instead of the common_domain_name.

So if you plan to share an Bugzilla Instance with others I think you need a separate one and can not use the existing one.

So please do the following changes in /org.eclipse.mylyn.releng/multipass/roles/bugzilla/tasks/build_start.yml

• two times replace line
  BASE_URL: "{{ remote_domain_name }}" (line 95 and 129)
  with
  BASE_URL: "{{ common_domain_name }}"
• serviceLabel: '{"type":"{{ bugzilla_type }}","url":"/{{ act_item.servmame }}/", "version": "{{ bugzilla_version.stdout }}", "info": "{{ bugzilla_info }}", "properties": { "host-name": "{{ remote_domain_name }}", "host-ip": "{{ansible_ssh_host}}", "ansible-name": "{{ inventory_hostname }}", "docker": "true" {{ extra_info_label }} } }' (line 55)
  should be
  serviceLabel: '{"type":"{{ bugzilla_type }}","url":"/{{ act_item.servmame }}/", "version": "{{ bugzilla_version.stdout }}", "info": "{{ bugzilla_info }}", "properties": { "host-name": "{{ common_domain_name }}", "host-ip": "{{ansible_ssh_host}}", "ansible-name": "{{ inventory_hostname }}", "docker": "true" {{ extra_info_label }} } }'

but even without the changes I can work with https://mylyn.local/bugzilla-4_4/ and did not find an place where the wrong urlbase or sslbase cause an error.

[gnl42]

Something has be to be missing or I would be able to access the VM using the external IP address

Frustrating.

[BeckerFrank]

one more time! Normally the remote_domain_name is not required, the common_domain_name is always sufficient.

You can you access https://mylyn.local/bugzilla-4_4/ or https://mylyn.local/mylyn_idx/ and do not have the need to use https://mylynmac.glindholm.me/bugzilla-4_4/ or https://mylynmac.glindholm.me/mylyn_idx/

[gnl42]

Again, the CI server is not my development machine which is why I need the remote_domain_name setup to work for both my main desktop and my laptop.

Ideally I would like to have it running on my pi5 box in the end

[BeckerFrank]

I create #583 for this. My use case did not cover this and I found an error so that the multipass vm did not get the additional IP Address.

[gnl42]

So I've got an mylyn.local environment up and running and I'm working on conditionally re-enabling the CI tests.

I've had a couple of issues come that I've fixed, but there are five Bugzilla tests that fail and I don't know why.
I'm not sure if I should disable the tests for the moment or keep trying to figure out what the issue might be.
Is it possible the mylyn.local data doesn't match the mylyn.org data?

[BeckerFrank]

Please let me know which tests have failed and I will look into it. But it may take some time because I have to take care of my wife's palliative treatment.

Maybe BugzillaHarness has not created a Task correctly.

[gnl42]

Sorry to hear about your wife Frank.

These are the failres:

Failures: 

  testGetTaskData(org.eclipse.mylyn.bugzilla.tests.BugzillaRepositoryConnectorTest): null


Errors: 

  testStdWorkflow(org.eclipse.mylyn.bugzilla.tests.BugzillaRepositoryConnectorTest): Task data at "E:\workspaces\mylyn-main\git\org.eclipse.mylyn\mylyn.tasks\connectors\bugzilla\org.eclipse.mylyn.bugzilla.tests\target\work\data\.metadata\.mylyn\tasks\bugzilla-https%3A%2F%2Fmylyn.local%2Fbugzilla%2D4%5F4\offline\137.zip" not found

  testAliasRetrieval(org.eclipse.mylyn.bugzilla.tests.BugzillaRepositoryConnectorTest): alias in use: Bug45 has already taken the aliasFritz. Please choose another alias.

  testAliasMultiRetrieval(org.eclipse.mylyn.bugzilla.tests.BugzillaRepositoryConnectorTest): alias in use: Bug45 has already taken the aliasFritz. Please choose another alias.

No rush

George

[gnl42]

Hi @BeckerFrank, thanks for all the work getting the CI setup straightened out.

Question for you. when I click on any of the bugzilla-5.2 links on the mylyn_idx page I get:

Software error:

The ./data/params.json file does not exist. You probably need to run checksetup.pl. at Bugzilla/Config.pm line 341.
Compilation failed in require at /var/www/html/index.cgi line 15.
BEGIN failed--compilation aborted at /var/www/html/index.cgi line 15.

Should I be seeing that?

Thanks
George