Issue #372: Accessing None in validate_koji_tag - fm-orchestrator

fm-orchestrator

#372 Accessing None in validate_koji_tag

Closed 7 years ago Opened 7 years ago by jkaluza.

Callstack that lead to the logging statement

  File "/usr/lib64/python2.7/threading.py", line 784 in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib64/python2.7/threading.py", line 811 in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 764 in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/site-packages/twisted/python/threadpool.py", line 167 in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 118 in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 81 in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.7/site-packages/moksha/hub/api/consumer.py", line 191 in _work
    self.consume(message)
  File "/usr/lib/python2.7/site-packages/module_build_service/scheduler/consumer.py", line 129 in consume
    log.exception('Failed while handling {0!r}'.format(msg))
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/module_build_service/scheduler/consumer.py", line 127, in consume
    self.process_message(session, msg)
  File "/usr/lib/python2.7/site-packages/module_build_service/scheduler/consumer.py", line 187, in process_message
    further_work = handler(conf, session, msg) or []
  File "/usr/lib/python2.7/site-packages/module_build_service/scheduler/handlers/components.py", line 118, in complete
    return _finalize(config, session, msg, state=koji.BUILD_STATES['COMPLETE'])
  File "/usr/lib/python2.7/site-packages/module_build_service/scheduler/handlers/components.py", line 80, in _finalize
    parent.owner, module_name, config.system, config, tag_name=tag)
  File "/usr/lib/python2.7/site-packages/module_build_service/builder.py", line 137, in create
    config=config, **extra)
  File "/usr/lib/python2.7/site-packages/module_build_service/utils.py", line 829, in wrapper
    if all([t.startswith(pre + allowed_prefix + post) for t in tag_list]):
AttributeError: 'NoneType' object has no attribute 'startswith'

Metadata Update from @fivaldi:
- Issue assigned to fivaldi

7 years ago

jkaluza commented 7 years ago

This is probably caused by PDC. It seems PDC is down or has some network issues and the result is that koji_tag is not generated fast enough and when we ask for it, our pdc.py code returns None for koji_tag.

So there are multiple parts of this issue:

1) When koji_tag is None, validate_koji_tag should handle that.

2) When validate_koji_tag raises, it should mark the module build as FAILED. We should probably make some general code which would be called for every message with the reference to module to mark the build as FAILED on exception.

EDIT: This could be handled globally with something like that (not working code, just to show the idea):

diff --git a/module_build_service/scheduler/consumer.py b/module_build_service/scheduler/consumer.py
index 8c6103c..848e2a6 100644
--- a/module_build_service/scheduler/consumer.py
+++ b/module_build_service/scheduler/consumer.py
@@ -170,10 +170,15 @@ class MBSConsumer(fedmsg.consumers.FedmsgConsumer):
         # Choose a handler for this message
         if type(msg) == module_build_service.messaging.KojiBuildChange:
             handler = self.on_build_change[msg.build_new_state]
+            build = models.ComponentBuild.from_component_event(session, msg)
+            if build:
+                build = build.module_build
         elif type(msg) == module_build_service.messaging.KojiRepoChange:
             handler = self.on_repo_change
+            build = models.ModuleBuild.from_repo_done_event(session, msg)
         elif type(msg) == module_build_service.messaging.MBSModule:
             handler = self.on_module_change[module_build_state_from_msg(msg)]
+            build = models.ModuleBuild.from_module_event(session, msg)
         else:
             log.debug("Unhandled message...")
             return
@@ -184,7 +189,12 @@ class MBSConsumer(fedmsg.consumers.FedmsgConsumer):
             log.debug("Handler is NO_OP: %s" % idx)
         else:
             log.info("Calling %s" % idx)
-            further_work = handler(conf, session, msg) or []
+            try:
+                further_work = handler(conf, session, msg) or []
+            except e:
+                if build:
+                    build.transition(config, models.BUILD_STATES['failed'], str(e))
+                    session.commit()
             log.info("Done with %s" % idx)

             # Handlers can *optionally* return a list of fake messages that

3) We still have the race-condition between the MBS and PDC. On module submission, MBS sends "init" message, which is handled by the pdc-updater to populate the koji_tag in PDC. In the meantime, MBS creates entries in database, parses the modulemd and so on. When MBS is done, it changes the module state to "wait" and it expects that there will be our module in PDC... But In case PDC is down or simply pdc-updater takes more time than MBS, we end up with broken module build, because the module entry is not yet in PDC...

Feel free to create multiple issues out of that, if you think it makes sense.

Edited 7 years ago by jkaluza

fivaldi commented 7 years ago

See PR#374.