#7563 Request for resources: Message-Tagging-Service
Closed: Fixed 4 years ago by bowlofeggs. Opened 5 years ago by cqi.

Message-Tagging-Service, aka MTS, is an event-driven microservice to tag a module build triggered by MBS event mbs.build.state.change.

MTS basically listens on message bus for the MBS event mbs.build.state.change. Once a message is received, the module build represented by that message will be tested if it matches any predefined rules. Each rule definition has destination tag defined. If a rule matches the build, the destination tag will be applied to that build. Only module build in ready state is handled by MTS.

MTS has been containerized and deployed in production in internal, so we would like to deploy it to Openshift as well to serve module build tagging in Fedora.

EDIT:

MTS requires a keytab to log into Koji and permission to tag build by calling XMLRPC API tagBuild.

Phase I

Phase II

Phase III

  • SOP link:
  • Audit request:
  • Audit timeline:

Phase IV

  • Ansible playbooks:
  • Fully rebuilt from ansible:
  • Production goal:
  • Approved audit:

Metadata Update from @bowlofeggs:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: mbs, request-for-resources

5 years ago

I've heard from out-of-band sources that there are questions about what problems the MTS solves.

The most clear-cut one is this: a module that is built once and run on multiple releases. For example, let's consider the Golang case. Many Golang packages are statically-linked (as is the custom of that ecosystem), so they have few or no external dependencies. However, let's say that this module only currently builds for Fedora 30, because it requires Golang 1.12beta at build-time. Once built, there's nothing preventing it from running on Fedora 28 and 29, so we'd define its dependencies in the module metadata as:

dependencies:
  - buildrequires:
      platform: [f30]
    requires:
      platform: []

This translates to "Build against Fedora 30, run on any available platform". Now, however, we need some mechanism to ensure that the built module is tagged into each of the supported platforms, not just into the one against which it was built. Today, this is a manual process whereby the maintainer does a build and then pings release-engineering to tag it manually into the appropriate releases. The module tagging service will set up rules to have this happen automatically.

Makes sense to me, @sgallagh. We should make sure @rbarlow is aware of this thought process, since Bodhi would need to promote the same module build through multiple tag hierarchies for that to be useful. Initial tagging by MTS here would be the first step.

On Thu, 2019-02-14 at 21:15 +0000, Ralph Bean wrote:

Makes sense to me, @sgallagh. We should make sure @rbarlow is aware
of this thought process, since Bodhi would need to promote the same
module build through multiple tag hierarchies for that to be useful.=20
Initial tagging by MTS here would be the first step.

Note that my management chain has decided that doing this is not a
priority right now. It requires a significant backwards incompatible
change to Bodhi, and we already have other large initiatives in
progress right now.

The Bodhi change (that would make it possible to have one build be included in multiple updates for multiple releases) should not be dependency for thins RFR as MTS is useful even without Bodhi change.

I will sponsor this RFR. With that we can move to phase 2, in which you should start a thread on infrastructure mailing list. See RFR SOP for details what information you should post to the list.

Metadata Update from @mizdebsk:
- Issue assigned to mizdebsk

5 years ago

Ticket releng#8176 is created to ask for agreement on this proposal.

Just to confirm that I agree to be a Development and Maintainership contact.

ack on being listed under Development and Maintainership.

I believe that we are ready to move out of planning phase - all questions on infrastructure list have been answered and release engineering was in favour of running MTS service in Fedora.

In the next phase a SOP document will need to be created. Please open a pull request for infra-docs adding relevant SOP document.

I think that the RFR part about creating a development instance can be skipped as we generally don't set them up for OpenShift apps (as far as I know).

Since MTS is closely related to MBS, can we re-use sysadmin-mbs as group of people allowed to administer MTS? sysadmin-mbs group is already used for various modularity-related services besides MBS.

@puiterwijk Can we schedule a security audit for this service, as required by the RFR process?

Since MTS is closely related to MBS, can we re-use sysadmin-mbs as group of people allowed to administer MTS? sysadmin-mbs group is already used for various modularity-related services besides MBS.

Hi @mizdebsk , from where can I see the members of group sysadmin-mbs?

Since MTS is closely related to MBS, can we re-use sysadmin-mbs as group of people allowed to administer MTS? sysadmin-mbs group is already used for various modularity-related services besides MBS.

Hi @mizdebsk , from where can I see the members of group sysadmin-mbs?

https://admin.fedoraproject.org/accounts/group/view/sysadmin-mbs

Thanks. @mizdebsk , can you please also add lucarval and vmaljulin into group sysadmin-mbs? Then, let's re-use this group.

The SOP PR looks good to me - I don't have permission to merge it myself.

@jkaluza @mprahl @ralph : Do you agree with using sysadmin-mbs group for controlling who has access to running MTS playbook? If yes, can you add @lucarval and @vmaljulin to that group?

I believe we are ready to move to the next phase - creating staging instance in OpenShift. @cqi Please let me know if you need any help with that.

@mizdebsk So far, there is no security issue reported by puiterwijk. Does it mean MTS have passed the security audit? Or, do we need to get confirmation from him about that?

I believe we are ready to move to the next phase - creating staging instance in OpenShift. @cqi Please let me know if you need any help with that.

I think yes. @mizdebsk This is my first time to go through these steps to deploy a service into Fedora infra. I have no idea if I could have permission to create a stage instance, from where and how to create. Or, are you the right person to help to create the stage instance?

I learned from the SOP Staging Instance step, I need to write ansible role to deploy MTS, but I'm not sure once the stage instance is created, what resources will be available for me to run the ansible playbook and test the deployment, proper permissions to ssh into stage host and run playbook, stage hostname that I can ssh, or something else? Do I need to file another ticket separately to Fedora infra to request permissions?

@cqi, the reason for no security items reported is most likely that @puiterwijk is currently tasked with a lot of other items which management have rated as higher priority. I do not know when he will be able to get get to this, but @jperrin would be the one to contact on priorities.

On the second point, you will need to create a playbook and sysadmin-main will have to add it to the rbac-ansible rules for you to run. You are in sysadmin-mbs so we would put it in that role. Please open a separate ticket on that

@smooge Thanks for the info. With some guide from colleagues, I succeeded to clone the infra ansible repository. I saw there is patches sent to Fedora infra mailing list, just wanna confirm if that is the normal way to request review and merge for MTS?

Hi @kevin, I'm writing mts role and I need to put a URL of a rule file into configmap.yml. Is there already an existing repo containing a rule file? Do I need to file a ticket for creating such a repo and the initial rule file?

I think that for now you can store the rule file directly in ansible.git. If you want to store it in a separate repository then you can create it yourself, eg. on pagure.io

@cqi What is the progress here? Do you need any help with creating repository for MTS configuration? Or with anything else (Ansible playbook, OpenShift objects etc)?

Am I right in thinking this functionality will also facilitate build-only modules?

In the sense that I have a module that is intended only for building other modules and is not meant to ship to users. However right now if I build that module for rawhide it will ship to users because there is no bodhi gating, is that correct?

Right now, when you build a module on rawhide, MBS will tag it by itself with f31-modular-updates-candidate Koji tag. Since F31 doesn't use Bodhi yet, builds tagged there are auto-signed and moved to f31-modular tag, from where they are composed and delivered to users.

After deploying MTS, MBS will no longer tag modules anywhere - this will be responsibility of MTS. Unlike MBS, MTS has a configurable policy that can express what should be tagged where. It should be possible to configure MTS in a way that it will tag buildroot-only modules differently, so that they won't be moved to f31-modular.

Hey @cqi any status here? Anything we can help you with to move this forward?

Hi @kevin , I'm just back from a long leave since the end of April. I will back to continue working on this issue again soon.

@kevin

Currently, there are some.

  • I would like to double-confirm that if only Kerberos authentication is allowed to login a Koji session in Fedora infra.
  • I forgot if I asked this question before, I'm still not clear about creating MTS in Openshift stg. Do I need to file a ticket for creating MTS in stg specifically? Or, am I allowed to create one by myself?
  • Similarly, what is the workflow to request MTS project in Openshift prod? Just a Fedora-infra ticket?
  • I followed doc[1] to have a look at Openshift stg. oc login fails, report "Login failed (401 Unauthorized)", and point me to obtain an API token from https://os.stg.fedoraproject.org:443/oauth/token/request. The token link shows "Application is not available" finally. Visit to https://os.stg.fedoraproject.org/ also shows the same issue. I'm not sure if this is normal nowadays for Fedora infra environment, or an actual issue.

[1] https://fedora-infra-docs.readthedocs.io/en/latest/dev-guide/openshift.html#openshift-in-fedora-infrastructure

I would like to double-confirm that if only Kerberos authentication is allowed to login a Koji session in Fedora infra.

Correct. Fedora Koji uses only Kerberos/GSSAPI authentication. SSL and password-based authentication is not allowed.

I forgot if I asked this question before, I'm still not clear about creating MTS in Openshift stg. Do I need to file a ticket for creating MTS in stg specifically? Or, am I allowed to create one by myself?

You don't need to open separate tickets. You need to come up with Ansible role and playbook for setting up MTS. The playbook will create OpenShift project. You can see examples in roles/openshift-apps/ and playbooks/openshift-apps/ directories in ansible.git. Let me know on IRC if you need any help with that.

Similarly, what is the workflow to request MTS project in Openshift prod? Just a Fedora-infra ticket?

Like above - run Ansible playbook which will create the project and deploy the application.

I followed doc[1] to have a look at Openshift stg. oc login fails, report "Login failed (401 Unauthorized)", and point me to obtain an API token from https://os.stg.fedoraproject.org:443/oauth/token/request. The token link shows "Application is not available" finally. Visit to https://os.stg.fedoraproject.org/ also shows the same issue. I'm not sure if this is normal nowadays for Fedora infra environment, or an actual issue.

This may be because you don't have your user created yet. Once the aforementioned playbooks are ran you should be able to login. I'll create minimal skeleton playbook to have the project set up. What project name do you want to choose? message-tagging-service, mts, or something else?

What project name do you want to choose? message-tagging-service, mts, or something else?

message-tagging-service, please. Thanks. :)

I've created message-tagging-service project in staging OpenShift.
Project web console can be accessed at https://os.stg.fedoraproject.org/console/project/message-tagging-service/
For now @mizdebsk and @cqi have access, more people can be added to "appowners" list by editing roles/openshift-apps/message-tagging-service/vars/main.yml file.
I've created initial openshift-apps/message-tagging-service.yml playbook and granted members of sysadmin-mbs FAS group permission to run the playbook
I've created Kerberos keytab which is mounted at /etc/krb5.keytab inside message-tagging-service container.
I've deployed bare service (with default configuration).
It is running in at https://message-tagging-service.stg.fedoraproject.org/

Now real configuration needs to be added and mounted at /etc/mts.
But I wasn't sure how the service should be configured, esp. messaging backend.
I think that use of AMQP ("rhmsg") backend would be preferred. Would that backend work with fedora-messaging?

Hi @mizdebsk, thank you very much for your help. I'll continue update the playbook. Regarding the messaging backend, we have an issue reported internally to migrate to fedora-messaging from fedmsg. Currently, I think we can just let MTS run with fedmsg firstly before the migration is done.

BTW, it should be a good idea to test rhmsg with Fedora AMQP infrastructure.

@mizdebsk, Hi, could you please give me the principal used to create keytab? It is required to be set in MTS app config.

@cqi you can use klist -kt /path/to/keytab to see the principals it contains.

@cqi note also that the principal is described by the playbook: https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/playbooks/openshift-apps/message-tagging-service.yml#n16

So the principal you want to configure is: message-tagging-service/message-tagging-service{{ env_suffix }}.fedoraproject.org@{{ ipa_realm }}

@mizdebsk Hi, I have updated ansible playbook and role and I'm going to apply changes to staging firstly to test. Am I allowed to push the changes to master branch directly?

@cqi Sorry, for some reason I didn't see your questions in this ticket until you pinged me on IRC.

  • Kerberos principal is message-tagging-service/message-tagging-service.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG, Kerberos keytab location is /etc/krb5.keytab
  • Yes, you can push required changes to master branch of ansible.git yourself (no reviews or pull requests are needed) and then run the following command on batcave01 host to apply the changes: sudo -i rbac-playbook openshift-apps/message-tagging-service.yml

@mizdebsk I succeeded to push my local changes to remote. But, I got an error when I run the command on batcave01:

TASK [openshift/project : Create project directory] *************************************************************************************************************************
Friday 28 June 2019  09:42:02 +0000 (0:00:00.124)       0:00:00.124 *********** 
fatal: [os-master01.stg.phx2.fedoraproject.org]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'app' is undefined\n\nThe error appears to be in '/srv/web/infra/ansible/roles/openshift/project/tasks/main.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Create project directory\n  ^ here\n"}

BTW, I ran the command successfully without -i, otherwise I got an error telling I'm not allowed to execute that command as root on that host.

@mizdebsk

Each time when I click Config Maps, I always get:

An error occurred connecting to the server.
Failed to list configmaps/v1 (status 403)

Anything wrong with my configmap.yml?

I ran oc get configmaps and got this:

Error from server (Forbidden): configmaps is forbidden: User "cqi" cannot list configmaps in the namespace "message-tagging-service": no RBAC policy matched

@mizdebsk

I got another forbidden error when I tried to rollout a new deployment with command oc rollout latest dc/mts -n message-tagging-service:

Error from server (Forbidden): deploymentconfigs.apps.openshift.io "mts" is forbidden: User "cqi" cannot update deploymentconfigs.apps.openshift.io in the namespace "message-tagging-service": no RBAC policy matched

@cqi All configuration and deployments of the application must be done through Ansible. Manual rollouts from CLI are not allowed, but you can add rollout to Ansible playbook, or set up a trigger in OpenShift. Viewing config maps is not allowed either as they can contain secret variables such as passwords.

@mizdebsk Thanks. So far, I've been testing MTS in stage now. I would like to double-confirm with you about the principal for stage, doesn't MTS principal have the part phx2? When using {{ env_suffix }} and running in stage, it is substituted with stg.phx2, then Koji API krb_login fails and I get error Server krbtgt/STG.PHX2.FEDORAPROJECT.ORG@STG.FEDORAPROJECT.ORG not found in Kerberos database.

Can I use principal message-tagging-service/message-tagging-service{{ env_suffix }}.fedoraproject.org@{{ ipa_realm }} for both stg and prod?

Like I said before, the principal is message-tagging-service/message-tagging-service.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG. This can be checked with ktutil tool like that:

$ ktutil 
ktutil:  rkt /etc/krb5.keytab
ktutil:  l
slot KVNO Principal
---- ---- ---------------------------------------------------------------------
   1    1 message-tagging-service/message-tagging-service.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG
   2    1 message-tagging-service/message-tagging-service.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG
   3    1 message-tagging-service/message-tagging-service.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG
   4    1 message-tagging-service/message-tagging-service.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG

env_suffix Ansible variable doesn't include phx2 part. It expands to ".stg" in staging and to empty string "" in production. Therefore yes, message-tagging-service/message-tagging-service{{ env_suffix }}.fedoraproject.org@{{ ipa_realm }} should be correct principal name to be used in Ansible templates.

Any update on when this will be available?

Edit: AFAICT, I can't build buildroot-only modules for Rawhide or Branched (until the Bodhi activation point) until this service is running without risking that users get packages they shouldn't.

I'm currently blocked by fedmsg config to publish message to fedmsg bus. The problem is fedmsg.publish(...) returns successfully, the message seems not be recorded in datagrepper. I've been testing MTS in stg environment. I tried some configs several times, but it looks none of them works for me.

Could anyone please point out the correct config for fedmsg to publish message to the bus? @mizdebsk Could you help please? Thanks!

@cqi This is a working configuration for fedmsg that greenwave used before moving to fedora-messaging --> https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/openshift-apps/greenwave/templates/greenwave.py?id=ab4db86c3c7dd7cb9c440fa66dce26dfba5bcb4d

I believe that for you to publish message you will need a certificate . Do you know if you have one ?

@cverna

I believe that for you to publish message you will need a certificate . Do you know if you have one ?

Oh, I don't know I need a certificate for publishing messages to fedmsg bus. By the way, is a certificate also required for fedora-messaging?

endpoints=dict(
   staging_gateway=[
       'tcp://stg.fedoraproject.org:9940',
   ],
),

In greenwave config, endpoints has a key named staging_gateway. For MTS, what key name should be? Is fedora-infrastructure usable or some other arbitrary name?

@cverna

I believe that for you to publish message you will need a certificate . Do you know if you have one ?

Oh, I don't know I need a certificate for publishing messages to fedmsg bus. By the way, is a certificate also required for fedora-messaging?

Yes you also need one for fedora-messaging.

endpoints=dict(
staging_gateway=[
'tcp://stg.fedoraproject.org:9940',
],
),

In greenwave config, endpoints has a key named staging_gateway. For MTS, what key name should be? Is fedora-infrastructure usable or some other arbitrary name?

I think the staging_gateway key is to make sure that we use the staging bus to listen to messages. Does MTS consume messages too ? if so I think you will need the same config in MTS.

Ticket fedora-infrastructure#8110 for requesting a certificate.

Ticket fedora-infrastructure#8110 for requesting a certificate.

Request is canceled since it is decided to migrate MTS to fedora-messaging, so that we only need to request a certificate only once for publishing messages to bus.

The deployment will continue after the migration is done. Thanks for your patience.

Hi, the resources for MTS has been available already. The certificate for fedora-messaging works for MTS. I think this issue could be closed. Thank you all for your help.

Metadata Update from @bowlofeggs:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

4 years ago

Login to comment on this ticket.

Metadata