fedora-qa

Clone
Source Code
GIT

#836 Migrate all Quality repos to Forgejo
Opened 3 months ago by adamwill. Modified 14 hours ago

Open
Close issue as:
Worksforme Fixed Wontfix Duplicate Invalid

We need to migrate all our repos to Fedora Forge - Quality.

Migration documentation: https://docs.fedoraproject.org/en-US/forge-documentation/migration/pagure/

Migration showstoppers:
b) There are milestones defined in the source repo. We're waiting for forge#281.
c) There are private tickets in the repo which need to be migrated. Private tickets can only be migrated to a separate private repository.

Migration checklist

  1. Look at showstoppers above, don't proceed for affected repos.
  2. Read migration documentation, in particular public repo migration and migrating dependencies and assignments
  3. When in doubt, test migration in Forge Staging first.
  4. Use the New Migration button in https://forge.fedoraproject.org/quality to start the migration
  5. Verify that everything has been migrated correctly (code, branches, tags, issues, pull requests, milestones, etc).
  6. Copy over the project URL for Forge manually. From Pagure project -> Settings -> Project Details -> Project's url into Forge project -> Settings -> Repository -> Website.
  7. Migrate assignees and dependencies. Verify that it worked correctly. Also inspect script output.
  8. Use forge_migrate_story_points.py to migrate story points. Verify that they it worked correctly. Also inspect script output.
  9. Use pagure_add_migration_comment.py to add a migration comment to all tickets in Pagure. Verify that they it worked correctly. Also inspect script output.
  10. In Pagure project -> Settings -> Project Options, enable Issue tracker read only and disable Pull requests.
  11. Replace the README contents of the repo with a message:
** This project has been moved to: $NEWURL ** 

This repository is now in archive mode only. There will be no further responses to any issues or pull requests in this repository. The development will only be taken forward in the new repository.
  1. (12.) If you use any kanban boards in the source repo, you need to re-create them manually in the new Forge repo.
  2. (13.) Go through all source code, internal and external documentation, RPM spec files, and any other locations, find references to the old repo and replace it with the new URL. (This might take a while, perhaps file a ticket for this so that you don't forget about it to do it later).
  3. (14.) If your new repo uses CI, enable it. You can get inspiration from Adam's howto blogpost, the fedfind repo and Forgejo Actions documentation.

Notes for migration

  • Project URL doesn't get migrated automatically, needs to be done manually. See forge#279.
  • Pagure tickets don't get a link or redirect to the new location. See forge#277. We've implemented our own solution, see the checklist.
  • Pull requests from forks are not migrated, i.e. you'll not have access to the forked branch. You need to pull from the fork and merge the changes manually.
  • Pagure boards (kanban) don't get migrated to Forgejo projects. See forge#280.
  • Pagure milestones have incorrectly set due dates. See forge#281 and forge#306.
  • Ticket assignees and dependencies (blocks/depends on) are not migrated by default, it needs an extra script execution. See forge#282 regarding Forge API token generation. I (kparal) tested the script and it works.
  • Our story_points field is not migrated. We've implemented our own solution, see the checklist.
  • When testing the migration to Forge staging, note that attachments might not appear properly, unless you source the repo from pagure.stg (ticket). But you can test the attachments by manually changing forge.stg to forge in the URL.
  • There are organization-wide labels and per-project labels. For the initial migration, we'll use per-project labels everywhere, but afterwards, we'll likely want to convert them in most projects to org-wide labels (some part of them, at least). New scripts will be needed for that.
  • Don't forget to enable email notifications in Fedora Forge for your account (they are disabled by default).

List of migration tickets for all Quality-related repos

[❔] https://pagure.io/fedora-qa/openqa_classifier/issue/1
[⚠️] https://pagure.io/fedora-qa/issue/839 BLOCKED by forge#281
[⚠️] https://pagure.io/fedora-qa/fedora_openqa/issue/114 BLOCKED by forgejo#10757
[⚠️] https://pagure.io/fedora-qa/os-autoinst-distri-fedora/issue/446 BLOCKED by forgejo#10757
[🚧] https://pagure.io/fedora-qa/blocker-review/issue/2009 BLOCKED by required engineering work
[✅] https://pagure.io/fedora-qa/check-compose/issue/3
[✅] https://pagure.io/fedora-qa/fedfind/issue/27
[✅] https://pagure.io/fedora-qa/python-wikitcms/issue/12
[✅] https://pagure.io/fedora-qa/relval/issue/48
[✅] https://pagure.io/issuebot/issue/16
[✅] https://pagure.io/fedora-easy-karma/issue/71
[✅] https://pagure.io/fedora-qa/createhdds/issue/16
[✅] https://pagure.io/fedora-qa/releasestream/issue/8
[✅] https://pagure.io/fedora-qa/python-ci_messages/issue/2
[✅] https://pagure.io/fedora-qa/relvalconsumer/issue/7
[✅] https://pagure.io/taskotron/resultsdb_conventions/issue/6
[✅] https://pagure.io/fedora-qa/testdays/issue/3
[✅] https://pagure.io/fedora-qa/qa-docs/issue/9
[✅] https://pagure.io/fedora-qa/qa-stats/issue/3
[✅] https://pagure.io/fedora-qa/qa-misc/issue/10
[✅] https://pagure.io/fedora-qa/testdays-web/issue/69
[✅] https://pagure.io/fedora-qa/openqa_testdata/issue/2
[✅] https://pagure.io/fedora-qa/fedora-release-autotest/issue/1
[✅] https://pagure.io/fedora-qa/blockerbugs/issue/295
[❌] autocloudreporter - OBSOLETE
[❌] autoqa - OBSOLETE
[❌] beaker-tools - OBSOLETE
[❌] commitfinder - OBSOLETE
[❌] fedora-beaker-tests - OBSOLETE
[❌] fedora-desktop-testing - OBSOLETE
[❌] fedora-kiwi-instance-docs - OBSOLETE
[❌] fedora-tcms-migration - OBSOLETE
[❌] fedora-testcases - OBSOLETE
[❌] landingpage - OBSOLETE (ticket)
[❌] lolz_and_roffle - OBSOLETE
[❌] modularity_testing_scripts - OBSOLETE
[❌] openqa_docker - OBSOLETE
[❌] switchcheck - OBSOLETE
[❌] testcase-playbook - OBSOLETE / unpopulated
[❌] uf-monitor - OBSOLETE
[❌] https://pagure.io/fedora-qa/openQA_scripting_tool/issue/1 - not needed now
[❌] https://pagure.io/fedora-qa/autococonut/issue/15 - not needed now
[❌] https://pagure.io/fedora-qa/fedora-glossary/issue/1 - not needed now
[❌] https://pagure.io/fedora-qa/test_cases/issue/2 - not needed now
[❌] https://pagure.io/fedora-qa/flask_skeleton/issue/2 - not needed now
[❌] cloud-atomic-ansible-tests - OBSOLETE
[❌] fedora-gooey-karma - ARCHIVED / OBSOLETE
[❌] kanban - OBSOLETE
[❌] motd - OBSOLETE
[❌] phabarchive - ARCHIVED; probably not needed since we have https://fedorapeople.org/groups/qa/phabarchive/
[❌] https://pagure.io/fedora-qa/_private/issue/37 - no longer needed, private issues are done in Jira now
[❌] taskotron-fedora-cloud-tests - OBSOLETE
[❌] windows-ad-integration-testing - probably OBSOLETE; we eventually implemented the 'AD' testing using Samba
[❌] qa-make - OBSOLETE
[❌] oraculum-client - NOT IMPLEMENTED
[❌] https://pagure.io/fedora-qa/oraculum/issue/221 - ORPHANED (ticket)
[❌] https://pagure.io/fedora-qa/packager_dashboard/issue/187 - ORPHANED (ticket)

Metadata Update from @adamwill:
- Custom field story_points adjusted to 5
3 months ago

Also we probably need an org in forgejo.

Yes, we need an org first. I was planning to look into it and figure out the required structure and naming and everything first. Feel free to beat me to it, if you wish.

The quality organization was created on both stg and prod deployments. Group mappings PR needs to get merged, and we need to fix one more issue with Anubis before we can move this forward. https://forge.fedoraproject.org/forge/forge/issues/264

Thanks for update, Tomas.

Another repo I'd like to migrate (and takeownership of) is
https://pagure.io/issuebot

I'll update the ticket description, add it, and convert it into checkboxes.

Metadata Update from @kparal:
- Custom field story_points adjusted to 13 (was: 5)
- Issue set to the milestone: Fedora 44
- Issue tagged with: meta, task
2 months ago

I've split "migrate this repo" into #839 and converted this ticket into an "Epic" to "migrate all QA repos".

I started editing the main description and adding helpful links and notes into. Feel free to add more stuff to it that you find.

I've started working on dealing with some of the post-migration issues in #850.

The scripts are almost ready, see #850. We should be able to test out guinea-pig migrations to production next week.

How Forge labels work:
1. Org labels and repo labels are separate entities, they don't override each other. Even when named correctly, they are distinct, so you can have two "bug" labels assigned to a ticket, one of them being an org label and the other one a repo label.
2. The Forge migration script always creates all existing Pagure labels as repo labels, and then assigns them as repo labels only (even when org labels exist).
3. Using API, I can retrieve both kinds of labels using GET and distinguish them (label name is the same, but the url field can be used to distinguish them), but I can't seem to be able to determine what label is assigned using POST. I give just a label name in POST, and it assigns whatever label (org or label) exists. If both exist, it assigns both labels.

This means:

  • The Forge migrator can be used right away, it doesn't interact with org labels (which we don't have ready yet).
  • It should be possible to covert repo labels -> org labels later using our tooling, by reading an issue, assigning the same label name (it will also assign an org label of the same name), and once all tickets are done, undefining the repo label from Forge UI (that will delete it from all issues).
  • For story points conversion (see #850), we need to have org labels ready first.

I've written a migration checklist (see this ticket description). I'll probably update all individual tickets with a reference to it.

I've also started a discussion on shared labels across the whole CLE, but since that discussion doesn't seem to be progressing a lot, I simply created the "Scrum" set on the Quality org level. Which means we can now migrate story points, because the points/* labels exist. We can figure out what to do with other labels later.

fedora-easy-karma has been migrated as the first project.

I found a serious problem in migration - issue attachments are not migrated properly. Let's freeze all migrations until it is resolved:
https://forge.fedoraproject.org/forge/forge/issues/321

I fixed that one over the break, I believe. Please check. I did notice another issue while working on it - https://codeberg.org/forgejo/forgejo/issues/10472 - but I don't think that one really needs to block the migration.

I've also experimented with Actions on staging forgejo and more or less have it working, see https://forge.stg.fedoraproject.org/quality/fedfind/pulls/30 . So we should be able to handle migrations of repos with CI as well once we get a runner in prod, I've requested one.

I fixed that one over the break, I believe. Please check.

Great, it works. I've updated the ticket description. We can now proceed with the migrations.

I did notice another issue while working on it - https://codeberg.org/forgejo/forgejo/issues/10472 - but I don't think that one really needs to block the migration.

I noticed it as well, but decided to ignore it as insignificant.

So we should be able to handle migrations of repos with CI as well once we get a runner in prod, I've requested one.

Thanks!

Did a bunch of mine today. Left fedora_openqa because the migration script chokes on it and os-autoinst-distri-fedora because it's huge and scary. Will see if the migration script can cope with o-a-d-f at all tomorrow, I guess, and maybe do some of the shared ones like qa-misc and qa-stats. I also re-arranged the list to be grouped by migration status, which I think makes it easier to work with. Please move entries to the appropriate place when changing status.
Edited 11 days ago by adamwill

We're down to 5 unmigrated Pagure repos. Two blocked by migration bugs, one to be migrated by Adam, one to be decided by Adam, and one that needs Forge support implemented in Blockerbugs.

Log in to comment on this ticket.

Metadata
None

task meta

None
normal
13
Powered by Pagure 5.14.1
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.