Migrate from Buildbot to GitLab

This is task for discussions/planning of migration, other more specific tasks may stem from this.

Final result should be that all Mentat related repos from Buildbot (and some external) should be migrated to GitLab, with adapted CI stage, pip upload and deb generation.

assigned to @rajmund_hruska

By Pavel Kácha on 2024-12-26T14:07:25

added C-Dev-Tools Done 0% P4 S-New T-Task labels

changed milestone to %Backlog

Relation 'relates' to: 7718

Relation 'relates' to: 7719

By Pavel Kácha on 2021-11-18T09:54:15

From discussion with Martin Dušička:

• There are 4 runners for general use directly on GitLab server
• GitLab Pages not working yet, Martin has to decide where (and how) to make them accessible
  • Pages would allow to publish docs and probably repos
• Wiki will be migrated as is, we will cleanup later (so we will have history on GitLab)

By Pavel Kácha on 2021-11-18T09:58:18

Question is how to manage PGP keys for deb package signing. Possibilities (already sorted by viability):

1. Extra repo with keyring and limited access (can be common for more projects, we don't mind binary contents as it gets changed rarely and never edited by hand)
2. Private and public key in GitLab variables (signing script must always reinitialize all new keyring, must be in variables in every project concerned)
3. Docker image with ready keyring (problem of encrypted contents on public place or need to keep own image resource, cumbersome and nonversioned updates of the keyring)

First one seems plausible.

By Pavel Kácha on 2021-11-18T10:06:47

I think I managed to make signing work on typedcols repository using the first option. I will just polish some things and then I will create the configuration file for other repositories.

There is one thing which I don't really understand. I tried verifying gpg signature of files at typecols Version 0.1.13 is working just fine but verifying previous 2 versions results in BAD signature and I don't know why.

By Rajmund Hruška on 2021-12-08T15:40:53

added S-Feedback label and removed S-New label

Rajmund Hruska wrote in https://gitlab.cesnet.cz/709/migrace/homeproj/mentat-project-05/mentat-ng/-/issues/7539#note_200549:

There is one thing which I don't really understand. I tried verifying gpg signature of files at typecols Version 0.1.13 is working just fine but verifying previous 2 versions results in BAD signature and I don't know why.

So, seems like for some older versions the files were replaced but the signatures weren't and that's why the verification of signature failed and the sha256sums were different.

Next thing to figure out is publishing documentation and signed files. So we are waiting until GitLab Pages is set and running.

By Rajmund Hruška on 2021-12-09T18:58:31

added Done 10% S-Waiting labels and removed Done 0% S-Feedback labels

The GitLab Pages are finally set. The next thing to figure out is how to publish signed packages.

By Rajmund Hruška on 2022-01-27T05:37:36

added S-In Progress label and removed S-Waiting label

So, I did manage to make both documentation and signed files publicly available. I am still working on how to share the cached files between branches which I thought I had already done, but it's not quite working yet.

Here are the example Pages for Pynspect project, master branch.
https://709.gitlab-pages-test.cesnet.cz/hruska/pynspect/master/html/manual.html
https://709.gitlab-pages-test.cesnet.cz/hruska/pynspect/master/files/

By Rajmund Hruška on 2022-02-11T07:42:39

added Done 50% label and removed Done 10% label

changed milestone to %2.10

Pynspect, Pyzenkit, typedcols, ipranges and idea are already at gitlab.cesnet.cz and CI/CD seems to be working. I only changed the version of Pynspect, all other projects have the same version number as before.

I didn't finish Pydgets yet, as I suspect the code in GitHub repository is not up to date. I wrote an email to @Jan_Mach regarding this repository.

https://709.gitlab-pages.cesnet.cz/idea/ipranges/master/html/manual.html
https://709.gitlab-pages.cesnet.cz/idea/idea/master/html/manual.html
https://709.gitlab-pages.cesnet.cz/idea/typedcols/master/html/manual.html
https://709.gitlab-pages.cesnet.cz/mentat/pynspect/master/html/manual.html
https://709.gitlab-pages.cesnet.cz/mentat/pyzenkit/master/html/manual.html

https://709.gitlab-pages.cesnet.cz/idea/ipranges/master/files/
https://709.gitlab-pages.cesnet.cz/idea/idea/master/files/
https://709.gitlab-pages.cesnet.cz/idea/typedcols/master/files/
https://709.gitlab-pages.cesnet.cz/mentat/pynspect/master/files/
https://709.gitlab-pages.cesnet.cz/mentat/pyzenkit/master/files/

By changing master to devel in the URL, it is possible to access the development documentation and signed files.

So apart from Pydgets, the next step is Mentat repository.

By Rajmund Hruška on 2022-04-17T10:57:35

added Done 60% label and removed Done 50% label

Pydgets is now also ready.

All of those smaller projects used simple python docker image. For Mentat it would be nice to have a better Docker image. Maybe this one could do the work, although it doesn't contain PostgreSQL. https://hub.docker.com/r/nikolaik/python-nodejs

By Rajmund Hruška on 2022-05-03T07:42:11

Wouldn't it make sense to make our own and push it to the hub? Seems creating Docker images based on pre-existing ones needs might need just short Dockerfile.

https://linuxhint.com/automatically-build-docker-images-in-debian-10-buster/

By Pavel Kácha on 2022-05-04T10:29:28

changed milestone to %2.11

I lost my notes from the meeting where we discussed this issue. So, I should create a job which downloads all the files at a given URL address? If I remember correctly, we want to get the older files from https://alchemist.cesnet.cz/pyzenkit/files/development/ and also have some kind of backup if we lose data from GitLab.

By Rajmund Hruška on 2022-09-21T09:52:38

added S-Feedback label and removed S-In Progress label

Rajmund Hruska wrote in https://gitlab.cesnet.cz/709/migrace/homeproj/mentat-project-05/mentat-ng/-/issues/7539#note_200556:

I lost my notes from the meeting where we discussed this issue. So, I should create a job which downloads all the files at a given URL address? If I remember correctly, we want to get the older files from https://alchemist.cesnet.cz/pyzenkit/files/development/ and also have some kind of backup if we lose data from GitLab.

We wanted to solve two main problems:

• Loss of continuity – something in Gitlab goes awry or human makes an error, and we thus lose the continually reloaded artifact containing older releases
• Migration of older data from Redmine

If memory serves me well (sorry, too much parity errors these days), we came to rather simple sidestep – define GitLab variable, which, if set to a bunch of URLs, would cause build script to download contents of all the URLs and populate the "history" artifact (or maybe one URL with expected zipfile? Just what's simpler). That would allow to inject initial history (from Redmine), which we are not able to reconstruct, and also to possibly inject history from backups if data end up in a blue smoke.

Also, we should actively backup those released files, for example by regular download to backed tree at Mentat-alt or somewhere.

By Pavel Kácha on 2022-09-21T13:16:42

Pavel Kácha wrote in https://gitlab.cesnet.cz/709/migrace/homeproj/mentat-project-05/mentat-ng/-/issues/7539#note_200557:

Also, we should actively backup those released files, for example by regular download to backed tree at Mentat-alt or somewhere.

So, the backup is set up on mentat-alt. If I understand it correctly I can keep the built packages in /var/backups/ .

By Rajmund Hruška on 2022-11-28T12:38:31

Rajmund Hruska wrote in https://gitlab.cesnet.cz/709/migrace/homeproj/mentat-project-05/mentat-ng/-/issues/7539#note_200558:

So, the backup is set up on mentat-alt. If I understand it correctly I can keep the built packages in /var/backups/ .

Yup, or some convenient subdirectory thereof.

By Pavel Kácha on 2022-11-29T14:29:45

added S-In Review label and removed S-Feedback label

The repository is created here: https://gitlab.cesnet.cz/713/mentat/mentat.

The next step is to set up mentat-alt to install mentat from deb packages created in GitLab.

By Rajmund Hruška on 2023-06-16T08:21:06

added Done 75% label and removed Done 60% label

From 2023-06-22 meeting:

• GitLab expects specific suffixes as a part of version.
• GitLab extracts version of the package from the changelog.

Our old packages (generated on Buildbot) don't satisfy neither GitLab requirement - although we have the mechanism to upload old packages along with new generated ones, packages are in wrong branch with wrong (all the same) fabricated version.

To add insult to injury, our deb packages are de facto envelope around venv and pip install, and keep static URL to the Buildbot.

Conclusion - attempts to incorporate old packages onto GitLab along with new ones would take too much effort with inadequate results. Let's not try unify old and new packages and let's keep URL to Buildbot on the web as obsolete and keep running Buildbot (maybe without Alchemist) for some time (a year maybe?). And then ditch it altogether.

By Pavel Kácha on 2023-06-23T15:14:06

changed milestone to %Future

mentat-alt is now installed and upgraded from GitLab packages. There is no need to push to buildbot/alchemist anymore.

By Rajmund Hruška on 2023-08-22T10:31:02

By Pavel Kácha on 2023-08-22T11:02:51

Relation Relates - Bug 7718 Gitlab CI/CD doesn't push the built package to PyPI

By Rajmund Hruška on 2024-03-19T15:51:14

Relation Relates - Config 7719 In GitLab CI/CD GeoLite2 databases are downloaded for almost every job

By Rajmund Hruška on 2024-03-19T16:03:59

CI/CD and package/repo creation complete. Big thanks again.

By Pavel Kácha on 2024-04-16T08:58:59

added S-Closed label and removed S-In Review label

closed