#2650 policy test is not thread safe
Closed: Fixed 3 years ago by tkopecek. Opened 3 years ago by mikem.

While running some heavily parallel tag operations in testing, I encountered the following error:

File "/usr/share/koji-hub/kojixmlrpc.py", line 314, in _dispatch
    ret = koji.util.call_with_argcheck(func, params, opts)
  File "/usr/lib/python3.8/site-packages/koji/util.py", line 271, in call_with_argcheck
    return func(*args, **kwargs)
  File "/usr/share/koji-hub/kojihub.py", line 10795, in tagBuildBypass
    assert_policy('tag', policy_data, force=force)
  File "/usr/share/koji-hub/kojihub.py", line 9893, in assert_policy
    check_policy(name, data, default=default, strict=True, force=force)
  File "/usr/share/koji-hub/kojihub.py", line 9776, in check_policy
    result = ruleset.apply(data)
  File "/usr/lib/python3.8/site-packages/koji/policy.py", line 357, in apply
    self.lastaction = self._apply(self.ruleset, data, top=True)
  File "/usr/lib/python3.8/site-packages/koji/policy.py", line 330, in _apply
    check = test.run(data)
  File "/usr/share/koji-hub/kojihub.py", line 9740, in run
    raise koji.GenericError("encountered policy loop at %s" % self.str)
koji.GenericError: encountered policy loop at policy build_tag_mismatch

There was no possible loop in the policy. The loop detection code is not thread safe. It stores depth state in an instance variable of the PolicyTest instance. Since policy is parsed at startup, these instances are shared across threads.


Metadata Update from @mikem:
- Custom field Size adjusted to None

3 years ago

Metadata Update from @tkopecek:
- Issue set to the milestone: 1.25

3 years ago

Metadata Update from @tkopecek:
- Issue set to the milestone: 1.24 (was: 1.25)

3 years ago

Testing notes

I encountered the issue running parallel tag operations (well, clone-tag). The policy in my test instance involved a subpolicy.

To replicate this, I believe you will need:

  • run mod_wsgi in a multithreading configuration (should be the default)
  • use a subpolicy (one policy refers to another using the policy test) in one of your policies
  • trigger the policy on the hub in parallel repeatedly

Likely, you'll want to use either the tag policy or pkglist policy, as these are the easiest to trigger with a simple hub call. You can trigger the tag policy with tagBuildBypass, and you can trigger the pkglist policy with packageListAdd. I would recommend grouping a bunch of these into a multicall to increase the chances of overlap.

Metadata Update from @jcupova:
- Issue tagged with: testing-ready

3 years ago

Metadata Update from @mfilip:
- Issue tagged with: testing-done

3 years ago

Login to comment on this ticket.

Metadata
Related Pull Requests
  • #2651 Merged 3 years ago