#19 misdirected emails about assertion failures
Opened 6 years ago by rombobeorn. Modified 5 years ago

I'm getting emails saying "dist.abicheck FAILED" for various packages. After following the link I find that the error is a failed assertion:

https://taskotron.fedoraproject.org/artifacts/all/989b44ce-255d-11e9-a292-525400fc9f92/tests.yml/GtkAda-2.24.2-30.fc30.log

I understand an assertion failure in the ABI checker to mean that the ABI checker is defective. This should be reported to the maintainer of the ABI checker, not to the maintainer of the package that it tried to check. If these mails keep coming I'll be inclined to ignore all notifications about dist.abicheck, and then I won't notice if it finds an actual problem with one of my packages some day.

Possibly the outcome should be "ERROR" instead of "FAILED", in case "ERROR" is supposed to mean an error in the test itself. (The documentation at https://qa.fedoraproject.org/docs/libtaskotron/latest/resultyaml.html lists some allowed values, but makes no attempt to define their meanings.)

If the assertion is actually intended to point out some problem in the package, then it needs a completely different error message, because this doesn't tell me anything.


This needs to be discussed with libabigail developer, we just execute the tool. Can you please raise a ticket at https://sourceware.org/bugzilla/enter_bug.cgi?product=libabigail and link it here? Thanks.
CC @dodji

@rombobeorn thank you for reporting this issue.

There are indeed two issues here. The first issue is that abipkgdiff (the libabigail tool used by the ABI verifier task-abicheck) shouldn't crash (assert) on the gtkada package. I am looking at that issue right now.

The second issue is that whenever abipkgdiff fails, that condition should be clearly reported so that the package manager (you in this case) doesn't loose time looking at the logs just to realize that it's libabigail that is failing.

@kparal whenever abipkgdiff fails and we set the outcome of the test to ERROR, do you know if there is a way to notify @sinnykumari and myself (task-abicheck maintainers) rather than annoying package maintainers?

Thanks.

Looking at run_abipkgdiff.py, the outcome is set to PASSED, FAILED or NEEDS_INSPECTION. The code could be modified to set the outcome to CRASHED (in case of abipkgdiff crashes), but that's not possible to submit to ResultsDB at the moment. We're discussing allowing that, though. If we allowed that, you could set up a FMN rule to get notified. At this very moment, there's unfortunately no way to get notified.

In all cases, it might be a good idea to place a visible note into the task output log if abipkgdiff crashes, so whoever reads the log understands that the results are incorrect.

If you're interested in us adding the CRASHED outcome and you or @sinnykumari would adjust the run_abipkgdiff.py file (I don't feel like touching the error handling code, I don't like bitwise operations :)), please tell us.

Kamil P=C3=A1ral pagure@pagure.io a =C3=A9crit:

kparal added a new comment to an issue you are following:
` If you're interested in us adding theCRASHEDoutcome and you or @sinnykumari would adjust therun_abipkgdiff.py` file (I don't feel
like touching the error handling code, I don't like bitwise operations
:)), please tell us.

Sure, I would be glad if the CRASHED outcome could be added. I'll
update the run_abipkgdiff.py file, unless @sinnykumari beats me to it
:-)

Thank you Kamil.

--=20
Dodji

I filed https://pagure.io/taskotron/resultsdb/issue/127 to track adding CRASHED support to ResultsDB.

Taskotron-dev ResultsDB should now support CRASHED. I also adjusted libtaskotron documentation.

@dodji @sinnykumari You have the ball now. Please adjust the wrapper to detect crashes, thanks.

The change is now deployed even in production.

Log in to comment on this ticket.

Metadata