#9579 Getting ACCESS_REFUSED when accessing queue
Opened 9 days ago by jsztuka. Modified 3 days ago


Could you please clarify, what is causing following behaviour

Following were aquired from jenkins logs after start of the instance.

2021-01-12 07:44:49.064+0000 [id=80]    SEVERE  c.r.j.p.c.m.RabbitMQMessagingWorker#subscribe: Eexception raised while subscribing job 'cvp-co-metadata-trigger', retrying in 1 minutes.
com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - access to queue '7391ebfe7478da526fe6ac48242d965c' in vhost '/public_pubsub' refused for user 'fedora', class-id=50, method-id=10)
    at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:522)
    at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:346)
    at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:182)
    at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:114)
    at com.rabbitmq.client.impl.AMQConnection.readFrame(AMQConnection.java:672)
    at com.rabbitmq.client.impl.AMQConnection.access$300(AMQConnection.java:48)
    at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:599)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - access to queue '7391ebfe7478da526fe6ac48242d965c' in vhost '/public_pubsub' refused for user 'fedora', class-id=50, method-id=10)
    at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:66)
    at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36)
    at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:502)
    at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:293)
    at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:141)
Caused: java.io.IOException
    at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:129)
    at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:125)
    at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:147)
    at com.rabbitmq.client.impl.ChannelN.queueDeclare(ChannelN.java:968)
    at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.queueDeclare(AutorecoveringChannel.java:333)
    at com.redhat.jenkins.plugins.ci.messaging.RabbitMQMessagingWorker.subscribe(RabbitMQMessagingWorker.java:78)
    at com.redhat.jenkins.plugins.ci.messaging.JMSMessagingWorker.subscribe(JMSMessagingWorker.java:49)
    at com.redhat.jenkins.plugins.ci.messaging.RabbitMQMessagingWorker.receive(RabbitMQMessagingWorker.java:158)
    at com.redhat.jenkins.plugins.ci.threads.CITriggerThread.run(CITriggerThread.java:73)
2021-01-12 07:44:49.066+0000 [id=80]    WARNING c.r.j.p.c.m.RabbitMQMessagingWorker#unsubscribe: Exception occurred when closing channel: Unknown consumerTag

cvp-co-metadata-trigger job should be triggered by events(pull_request created, etc.) from upstream repository.

Fedora-messaging provider handles those messages from upstream.

Upstream repo : operator-framework/community-operators

Is is possible to get response before EOD 2021/01/14,
Thanks in advance.



Can you explain more about where this is and what it's trying to do?

Are you just trying to get messages? or also write them?

User 'fedora' sounds like the ro receiving user? but then it cannot make it's own queue's...

Metadata Update from @kevin:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: low-gain, low-trouble, ops

9 days ago

User 'fedora' sounds like the ro receiving user? but then it cannot make it's own queue's...

They can, in the public_pubsub: https://fedora-messaging.readthedocs.io/en/stable/quick-start.html#fedora-s-public-broker when queues that are UUID and have some restrictions.

The core of the stacktrace seems to be when shutting down after running into:

channel error; protocol method: #method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - access to queue '7391ebfe7478da526fe6ac48242d965c' in vhost '/public_pubsub' refused for user 'fedora', class-id=50, method-id=10)

Which certs do you use to connect?

We are using these certs to create keystore and truststore
wget https://raw.githubusercontent.com/fedora-infra/fedora-messaging/stable/configs/fedora-key.pem
wget https://raw.githubusercontent.com/fedora-infra/fedora-messaging/stable/configs/fedora-cert.pem
wget https://raw.githubusercontent.com/fedora-infra/fedora-messaging/stable/configs/cacert.pem
Furthermore testing connection from jenkins ends up with success:
Successfully connected to rabbitmq.fedoraproject.org:5671

Are those the ones shipped in the fedora-messaging RPM?

No I'm not sure if they are the same, I am only getting those from github, is there a way to check that they are the same?

I've just compared those we re getting from github and those from fedora RPM and they are identical.

Can you explain more about where this is and what it's trying to do?

Are you just trying to get messages? or also write them?

User 'fedora' sounds like the ro receiving user? but then it cannot make it's own queue's...

cvp-co-metadata-trigget is job to validate changes that are made in upstream GH - community-operator.
We are only trying to register the messages from fedora-messaging and trigger this job accordingly.

Is this possibly related to https://pagure.io/fedora-infrastructure/issue/9385 ? Are you specifying a queue name ? or ?

For queue name we generate random uuid, which is then set in job cofiguration itself.
And the configuration for the fedora-messaging is present on the global config of jenkins itself, where no queue is specified.
I dont think this issue is related with the problem we are having which is

Caused: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - access to queue 'f2a6e058cd17395c8a9a6783def50d5b' in vhost '/public_pubsub' refused for user 'fedora', class-id=50, method-id=10)

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog