Every time I've seen debuginfod mentioned on irc, it was in the context of this experience, with people asking how to to turn it off.

This bug has information on how to disable the debuginfod feature.
https://bugzilla.redhat.com/show_bug.cgi?id=2026402

I don't think this should have been accepted into Fedora in its current state.
For enabling it by default, it needs to
- have reasonable size limits, or
- download in the background, or
- be easy interrupted at the gdb prompt

Hmm, the limited tests I did when this feature was in early stages were positive - I don't remember any long interruptions except when I expected long interruptions, and feedback seemed appropriate.

I wonder if there's some particular pathology going on in this case. gdb has done lazy-loading of debuginfo as long as i can remember, so wh would be downloading the llvm debuginfo unless you are debugging into the shader compiler?

I think the first thing to do would be to reach out to Frank Eigler to see if things are working as expected.

I don't think this should have been accepted into Fedora in its current state.

I agree, but primarily only because progress reporting is broken. debuginfod enabled by default is critical for developer experience in order to remove the manual debuginfo installation step that used to be required. But currently it shows each debug package at -0.00 MB size. :(

For enabling it by default, it needs to
- have reasonable size limits, or

I don't think a size limit is needed. If a huge debuginfo is really needed, then waiting is better than producing a useless backtrace. Any size limit would make WebKit or LLVM bugs impossible to solve, for example.

Ideally it would only download debuginfo for frames that actually appear in the backtrace, rather than for every library that an application links to. I don't think that needs to be a blocker, though.

• download in the background, or

It's impossible. The debuginfo has to be downloaded before you try to use gdb, or else gdb is not useful.

• be easy interrupted at the gdb prompt

It actually is easily interrupted if you press Ctrl+C. I don't remember if that interrupts a single download, or the whole thing.

More unfortunate problems with debuginfod:

• It seems to be much slower than installing debuginfo with dnf, though this is anecdotal and uncertain. I think it may be downloading only one at a time? A safer default would be to download 10-15 packages at a time.
• If gdb stops to prompt you to press (c) to continue scrolling, debuginfod will stall and the download will fail. This means gdb cannot be left unattended while it is downloading packages.

I wonder if there's some particular pathology going on in this case. gdb has done lazy-loading of debuginfo as long as i can remember, so wh would be downloading the llvm debuginfo unless you are debugging into the shader compiler?

It's not a regression relative to dnf debuginfo-install, which is the best we had prior to F35. That would install all debuginfo for every library linked to the executable that you are debugging. I think it was less problematic because it seemed to go faster than debuginfod.

This is just broken and needs to be disabled

BTW I agree that as long as progress reporting (-0.00 MB) is broken, it's not ready for Fedora. I wonder if there exists a bug report for this? If not, we'll need to create one.

The gdb interaction was tweaked recently, and perhaps made worse than necessary. (The -0.00MB was just one problem.) Several of our developers are working on it as we speak. cc: @amerey

You may find $DEBUGINFOD_MAXTIME=NNN [seconds] setting convenient to set a short overall timeout for downloads. Other options are documented here:

https://fedoraproject.org/wiki/Debuginfod

• It seems to be much slower than installing debuginfo with dnf, though this is anecdotal and uncertain. I think it may be downloading only one at a time?

I am pretty sure that this is not the case. The difference is that now gdb can get ALL the debuginfo for a process easily, including all the transitive shared libraries. So it asks, and debuginfod satisfies. If you were to hand-install all the transitive -debuginfo packages yourself, pretty sure you'll be spending about least as much space/time downloading & installing them.

I'm hoping we get time to do https://sourceware.org/bugzilla/show_bug.cgi?id=27913 before long, which should help this in a different way.

If you were to hand-install all the transitive -debuginfo packages yourself, pretty sure you'll be spending about least as much space/time downloading & installing them.

dnf debuginfo-install seems to be able to mostly do this, though you have to run it twice, and it always fails for a couple random packages. That's what I'm comparing to.

It may be that it's not actually slower, but feels slower because the progress reporting is busted.

I don't think a size limit is needed. If a huge debuginfo is really needed, then waiting is better than producing a useless backtrace. Any size limit would make WebKit or LLVM bugs impossible to solve, for example.

I am not trying to produce a backtrace to upload somewhere. I am debugging my own gtk code, and I could not care less about llvm symbols.

And every time I do that, it ends up like this:

Downloading -0.00 MB separate debug info for /lib64/libLLVM-13.so
[                                                                                                                 ]

And then it just sits there until I switch to a different terminal and killall gdb. Control-C does not work.

I would describe that as 'hostile work environment' for developers

@mclasen apologies for the inconvenience this has caused.

A number of improvements such as notice indicating debuginfod will be used plus a fix for the ctrl-c bug made their way upstream some time ago but the backport ended up being complex and took much longer than expected.

The updated GDB package with these improvements is currently on the testing phase and should be available within a week or two. We are also working with upstream to fix "-0.00 MB" as soon as possible.

Thanks for the updates, @amerey. Thats good to hear

The updated GDB package with these improvements is currently on the testing phase and should be available within a week or two. We are also working with upstream to fix "-0.00 MB" as soon as possible.

Any ideas how long it might take to release the updated gdb and fix "-0.00 MB"? The complaints about broken gdb in Fedora (and only Fedora) keep rolling in. I'm thinking we may need to back this out in F35 and try again in F36.

Any ideas how long it might take to release the updated gdb and fix "-0.00 MB"? The complaints about broken gdb in Fedora (and only Fedora) keep rolling in. I'm thinking we may need to back this out in F35 and try again in F36.

@catanzaro the gdb update is currently submitted for stable:

https://bodhi.fedoraproject.org/updates/FEDORA-2022-39ffce84e3

This update includes a fix for the ctrl-c bug and new commands that allow debuginfod to be enabled/disabled from gdb itself. It also adds a notice that prints when debuginfod is used for the first time indicating what it does and the user is prompted to enable/disable debuginfod.

As for upstream fixes I hesitate to offer an estimate. The scope of the patch has already grown based on reviewer feedback and it's likely there will be further iterations before the patch is merged.

FWIW if debuginfod is disabled by default until F36 this will affect other tools that use the service.

I see there is now a prompt to enable debuginfod:

This GDB supports auto-downloading debuginfo from the following URLs:
https://debuginfod.fedoraproject.org/ 
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.

However, if you hit y you still get -0.00 progress:

Downloading -0.00 MB separate debug info for /lib64/libjson-glib-1.0.so.0
Downloading -0.00 MB separate debug info for /home/mcatanzaro/.cache/debuginfod_client/ff150bb944b7d341fb33c367b230a0c005ac1833/debuginfo
Downloading -0.00 MB separate debug info for /lib64/libpipewire-0.3.so.0                                              
Downloading -0.00 MB separate debug info for /home/mcatanzaro/.cache/debuginfod_client/6ff1f3964dba6529ff3ebb093c3f78023e81311a/debuginfo
Downloading -0.00 MB separate debug info for /lib64/libgio-2.0.so.0                                                   
Downloading -0.00 MB separate debug info for /home/mcatanzaro/.cache/debuginfod_client/885a350082fb5150c77466144c54897321e3a93d/debuginfo

Proposal: revert this change for now in both F35 and F36, gdb developers to propose reenabling only once progress reporting is fixed.

Proposal: revert this change for now in both F35 and F36, gdb developers to propose reenabling only once progress reporting is fixed.

@catanzaro, upstream work on progress reporting continues but we are still not in a position to say when this work will be merged.

I support keeping debuginfod enabled in F35 and F36. Debuginfod support is available in various tools on Fedora beyond just gdb. Switching debuginfod to disabled-by-default could interrupt this functionality in all of these tools despite the service working as intended.

As for gdb debuginfod support, it does work as advertised: it makes debugging resources automatically available to the debugger and users avoid having to manually install these. Users are made aware of what debuginfod will do during their gdb session and they must first choose to enable it before it will download anything. Clear instructions are given for permanently disabling it if they so choose.

@catanzaro, upstream work on progress reporting continues but we are still not in a position to say when this work will be merged.

Hi, can you just disable the progress reports in the meantime, so it doesn't look broken?

Hi, can you just disable the progress reports in the meantime, so it doesn't look broken?

Hi, we discussed this at our meeting today. The Working Group would like you to disable the progress reporting in gdb until the -0.00 MB bug is fixed.

Feel free to reenable it when you've got it working properly, without any further approvals.

Hi, we discussed this at our meeting today. The Working Group would like you to disable the progress reporting in gdb until the -0.00 MB bug is fixed.

Feel free to reenable it when you've got it working properly, without any further approvals.

That shouldn't be a problem. We can remove the size and progress bar from each progress update and simply print

Downloading separate debug info for /lib/libxyz.so.0...

once at the start of each download.

That sounds good.

@catanzaro, the progress reporting fix has been added to gdb-11.2-2.fc35 and gdb-11.2-3.fc36 which are now available on F35 and F36. The fix is also in the process of being added to F37.

Excellent, thanks!

FWIW, I ran into more unpleasantness in debuginfod-client:
https://sourceware.org/bugzilla/show_bug.cgi?id=29022

Alas, reopening. I don't know what those files are for, but please make sure all files in the user's home directory are both readable and writable....

Alas, reopening. I don't know what those files are for, but please make sure all files in the user's home directory are both readable and writable....

A patch for this has been merged upstream and will be available on Fedora in the next update to the elfutils-debuginfod-client rpm.

@catanzaro what's the status of this issue? I see that you tagged it with pending-action...

This is fixed.