Update, another issue created by the AI, at least in this proposal:
7) the "_" in the wiki title have not been replaced by " " (spaces) in the subject title. This issue occurred only in the mailing list announcement, but not in Discourse.

Hi @py0xc3 I apologize for the inconvenience and inconsistencies with the processing of your change proposal. I take responsibility for the output and will do better reviewing content ahead of publishing. That being said, the AI-generated posts have been largely successful up until this point, so I put a bit of an assumption that "most was fine" with your proposal. I will make sure I do a more thorough review in upcoming change processes.

I also have some comments in-line below. Thanks!

1) The section "Current status" has been completely removed in the devel announcement mailing list AND in Discourse (this might be somehow provoked by my mistake to not add the targeted Fedora release version?)
--> this is by design in the documentation and training I received for processing changes.

2) The title of the Discourse topic and of the announcement in the mailing list is without release number ("Change Proposal: ... ...")
--> The Release Number is pulled from the Current Status section in the wiki (despite Current Status not being pulled in). This is due to a timing issue, not a code/AI issue. I had processed this change quickly after it was originally posted, just by chance. At the time I saw it, the Release had not been populated-- see https://fedoraproject.org/w/index.php?title=Changes/Restrict_ptrace_for_unprivileged_users_to_child_processes_to_match_kernel_default&action=edit&oldid=757398. The fact I processed it is due to a gap in understanding--it not being there did not raise a red flag to me. I thought it was left off by design.
3) The section "Early Testing (Optional)", which I did not use in my proposal (= I removed it from the template when drafting the proposal), was re-added by the AI in Discourse (content of the section added by the AI was "N/A"), BUT it was NOT added to the devel announcement mailing list
--> Thank you for the call out here. I have now updated the code to have the Discourse Discussion and Email draft match. It had not been obvious up until this point that the email draft was ignoring the sections whereas discourse was putting and N/A. For cleaner code/processing, empty sections will now always get an N/A. It is easier for pulling from the API.
4) The AI merged most paragraphs by removing the empty line in between paragraphs, but the "new line" has remained in these cases
5) Related to 4): The AI added several "\n" in the "Release Notes" section, which replaced each a "new line" so that a few paragraphs have been both merged, but also the "new line" have been removed in these cases -> so in this case, the respectively two affected paragraphs came together in one line. While I understand how correlations can cause the "\n" to happen, I wonder that it occurred only in the "Release Notes" section.
--> I apologize for this-- this did not do your proposal justice and with a lack of familiarity with md, I was not clear that this was an issue. However, I did review the code and output and made some changes to ensure the posts are more readable with line breaks.

6) Most formatting has not been adjusted from wiki to fit Discourse/mailings: while I am not sure if that was intended in the first place, I would say that if not, it might be easier and more predictable to just copy/paste from the wiki: even if the formatting ain't perfect, unexpected outcomes or changes can be excluded that way.
--> I think I addressed this in #4

7) the "_" in the wiki title have not been replaced by " " (spaces) in the subject title. This issue occurred only in the mailing list announcement, but not in Discourse.
--> This is the first time this has happened-- I should have noticed it before sending the email. The root cause has been resolved, but I apologize for the formatting in the announcement.

I take responsibility for the output and will do better reviewing content ahead of publishing

This is not about blame or so. It is OUR responsibility as a community to find an appropriate solution that is reliable for everyone without putting inappropriate burdens on individuals, in this case, put the burden of identifying+mitigating any AI-mistake on you, which I am not sure if it can be done reliably by any one individual given the different types of proposals we have. We have to provide some tools for you that work out fine for you and without unnecessary risks, or at least rationalize the risks contained and mitigate them.

That being said, the AI-generated posts have been largely successful up until this point, so I put a bit of an assumption that "most was fine" with your proposal

This is the problem what I meant with "unpredictable". Whenever an author makes something different, or uses code or commands or formatting in ways others didn't use before, it cannot be said for sure what the AI is doing with it. The same for constellations of text and/or code that were not deployed that way before. This risk exists at each proposal. And when it is about transferring commands or so from the wiki, minor changes that make a 0 to a 1 can be serious, and it can be said to be an inappropriate burden on you to review each case to this level of precision (which likely would cause more time than you can invest and likely more time than the AI seemed to save in the first place).

Effectively, you will never know what you have to look out for until it is identified, which makes it likely that unintended AI-actions (based on complex correlations that do not follow human reasoning) are overlooked. This is why I do not think you should be blamed for that (nor should you do this yourself), but it is also the reason why I am not sure if that solution is the best on the long term. But this place might be the best to shift this to a wider community and find out if anyone has applicable expertise to solve this :)

Beyond checking if anyone can create and maintain a python tool for them

I've been the kinda Unofficial Tool Guy for the Change stuff for the last couple of years. I do have this and various other things in the process that could be automated on my "that would be nice" list but haven't had any time for them yet.

Doing wiki -> markdown (for discourse) and wiki -> plaintext (for mailing list) conversion isn't really that difficult, if I could find a few hours I could probably knock something out. I kinda wanted to do more than just the text conversion bit if I was going to attack it, though, I wanted to get much further towards full automation, something like 'all the wrangler does is change the category and everything else happens magically'.

edit: I should also note i was dragging my feet a bit because I had the vague impression somebody was going to re-design the process overall. It does seem a bit odd that we managed to design a process which involves the wiki and mailing lists and discourse and bugzilla and at least two different issue trackers (fesco and release notes). You'd kinda figure we could slim that down a bit somehow.

Hi all,

For reference, the mode and documentation for all of the tools I have been developing can be found in this repo:https://gitlab.cee.redhat.com/caseteam/case/-/tree/main/Fedora%20Release%20Management%20Tool

Hope this helps bring some transparency to the process.

Most of us cannot see that repository, since it's behind the Red Hat firewall.

I don't like having changes being edited without the author's knowledge, especially if it's in subtle ways. From my perspective, you can decide to use AI to develop a tool to help you with the changes but in the end the program that process the changes should be deterministic, open source and publicly available for review.

So looking at the repo Alison referenced - Alison, please do put it up on pagure.io or forge.fedoraproject.org , Fedora tooling needs to be public and open source - it is a vibe-coded deterministic tool. It looks like...more or less what I would've written if I had the time, kinda. Except because it's vibe coded it's several thousand lines of code no human actually went through the process of writing, so no human has a map of how it works in their head.

Just poking through it I see a few obvious issues, like format_code_blocks is called in various places but is a complete no-op, and it's doing a lot of wiki text parsing which it would probably make sense to outsource to mwclient or another wiki library. More broadly, you'd want to look at just using pandoc to do to the conversion between wikitext, markdown-for-discourse and markdown-for-email - this kinda looks like it implements it all from scratch.

Overall it gives me the vague feeling a human could do it a lot more efficiently but it'd take a few hours evaluation to be really sure, and it's hard to say whether it's better to do that, just ignore it and start from scratch by hand, or keep on with the vibe coding approach and just keep feeding the LLM feedback of the kind in the first post here and hope it can find and fix the bugs in its own code.

I could ask my manager (@mattdm ) if I can ditch some of my other planned work and try to spend a few days working on this stuff (with whichever approach seems best) if that would help...I don't know if anyone else is in a position to work on it.

I don't like having changes being edited without the author's knowledge, especially if it's in subtle ways. From my perspective, you can decide to use AI to develop a tool to help you with the changes but in the end the program that process the changes should be deterministic, open source and publicly available for review.

BTW, to be fair, this was already happening, it was just being done by hand. At least when I was helping out with change wrangling that's how I did it - just copy/pasted the wikitext and reformatted it for email / discourse by hand. So that was "without the author's knowledge" and it's entirely possible a human might introduce subtle errors too. I think Aoife and Ben did the same, but if not I'm sure they can correct me...

Ben had his own scripts for this: https://pagure.io/fedora-change-wrangler

BTW, my suggestion for a better process - which would be much easier to automate - would be to ditch the wiki, and the full list emails. Make discourse the main place. Use discourse features as we do with common issues - use categories and tags for organizing the steps of the process and marking which release a Change is for. Don't send full text emails to the lists, just pointers to the discourse thread(s). FESCo and release notes issues could still exist, and we could automate their creation based on the discourse process, I guess.

Not sure whether we still need tracking bugs, maybe we could do the "progress check" part in discourse too. We might have to separate discussion from progress tracking for controversial changes I guess.

Ben had his own scripts for this: https://pagure.io/fedora-change-wrangler

The official ones that he passed on to Aoife are the ones at https://pagure.io/fedora-pgm/pgm_scripts/ , but they don't really have anything to handle the announcement part of the Change process.

Having a quick look through that fedora-change-wrangler repo you linked, it seems to be based around the use of Taiga, but we never actually went through with that, IIRC. It seems to have been part of https://fedoraproject.org/wiki/Changes/fedora-change-wrangler , which never quite happened . The upstream report links from that mail are now 404, not sure what happened there.

BTW, my suggestion for a better process - which would be much easier to automate - would be to ditch the wiki, and the full list emails. Make discourse the main place. Use discourse features as we do with common issues - use categories and tags for organizing the steps of the process and marking which release a Change is for. Don't send full text emails to the lists, just pointers to the discourse thread(s). FESCo and release notes issues could still exist, and we could automate their creation based on the discourse process, I guess.

Not sure whether we still need tracking bugs, maybe we could do the "progress check" part in discourse too. We might have to separate discussion from progress tracking for controversial changes I guess.

I already dislike interacting with Discourse for Changes, I definitely don't want to go whole hog into writing them directly into Discourse.

EDIT: To make this abundantly clear, I don't hate forums, I specifically dislike Discourse as a forum system.

BTW, my suggestion for a better process - which would be much easier to automate - would be to ditch the wiki, and the full list emails. Make discourse the main place. Use discourse features as we do with common issues - use categories and tags for organizing the steps of the process and marking which release a Change is for. Don't send full text emails to the lists, just pointers to the discourse thread(s). FESCo and release notes issues could still exist, and we could automate their creation based on the discourse process, I guess.

Not sure whether we still need tracking bugs, maybe we could do the "progress check" part in discourse too. We might have to separate discussion from progress tracking for controversial changes I guess.

I already dislike interacting with Discourse for Changes, I definitely don't want to go whole hog into writing them directly into Discourse.

With the Change process we effectively have effectively re-invented a workflow for code submission, code review, maintainer approval, all while going out of our way to avoid using the normal tools developers would for these tasks. IMHO we should have a git forge containing markdown documents for each change, a directory per Fedora release, proposals be pull requests which anyone can comment on, and FESCO then approve or reject. The git forges contain all the functionality needed for such an approach without having to write any custom scripts and abuse tools not designed for this kind of job.

After working with forge-based processes (like the one ALESCo uses) for this, I prefer the Fedora one because it isn't code-oriented. I like the wiki-style real-time editing model Fedora uses over the pull request model AlmaLinux uses.

That's not to say we couldn't do this with a git-backed thing, but I think the separation of writing/presentation, review, and acceptance is actually valuable. Most formal change processes have this separation because each stage has a different audience.

A first step would be to stop copy/pasting the content of the change across formats and instead link to the Wiki page only.

The git approach sounds really appealing as it would make it much easier to review changes to change requests, especially once accepted as right now, it's all hidden in the Wiki history and gets no review.

I would definitely be in favor of using the full git flow for reviews and discussions. Storing just the accepted changes in a git repo would already be a big improvement.

After working with forge-based processes (like the one ALESCo uses) for this, I prefer the Fedora one because it isn't code-oriented. I like the wiki-style real-time editing model Fedora uses over the pull request model AlmaLinux uses.

The wiki editting combined with copying to discourse is what I find to be particularly painful. We can be having a discussion about a change and no one realizes the change author has already updated the wiki with earlier feedback. We continue discussing problems that no longer exist in the wiki text, with no notifications that updates have been made.

That's not to say we couldn't do this with a git-backed thing, but I think the separation of writing/presentation, review, and acceptance is actually valuable. Most formal change processes have this separation because each stage has a different audience.

Most changes are fairly straightforward and don't receive huge volumes of comments, and I don't see a compelling need to have a bunch of separate tools for separate groups of people cobbled loosely together - IMHO that looses far more than it wins.

Where I admit a git forge review process could struggle though, is in the couple of Change proposals per Fedora dev cycle that trigger incredibly large comment threads. That's not to say email or discourse cope well with such large threads either, just that my gut is that git forges would cope even less well at that kind of volume of comments. I wonder if there's any reasonable way to allow 95% common case to happen entirely in a git forge, while still facilitating spin out into discourse for the handful of change proposals that warrant a free form debate ?

The process as it is now is not great, but it's mostly works. I think the issues OP listed are cosmetic, they are not something that requires that we change the process immediately.

We are very likely to change to the new forge before the next election cycle. So at least pagure will be out of the equation. I think it doesn't make sense for @alking or @adamwill to spend time on improvements to code when a lot of this will need to be redone in a few months anyway.

my suggestion for a better process - which would be much easier to automate - would be to ditch the wiki, and the full list emails. Make discourse the main place. Use discourse features as we do with common issues - use categories and tags for organizing the steps of the process and marking which release a Change is for.

Yes, I think this is the direction to go. Maybe we could even make use of the discourse mailing features to send a copy of the proposal to fedora-devel and fedora-announce. But I think we shouldn't take any action on this until after we have migrated the fesco repos to forgejo.

I think the issues OP listed are cosmetic

100%. The question I wanted to raise primarily is if we can exclude that an AI that adds sections, adds \n and removes lines does not make more problematic changes too, and if there are better solutions without too much efforts (and I think little people do review their proposals concerning if they had changed -> we might wouldn't know if changes had occurred more often).

The suggestions to stop copy/pasting but just link would effectively solve the problem and improving the workflow for Allison without much coding, just like the suggestion to centralize content in Discourse or any one media. With the changes that are to come, investing much time in coding indeed makes not much sense.

It seems we are now discussing two separate, but related issues, one being the use of AI to help generate code and improve the existing process for handling Fedora Changes, the second being an overhaul to the process of Changes Processing in general.

For the sake of the initial intent of this Fesco issue, I am going to focus only on the the use of AI in the existing process.

Based on the discussion here, I have moved my repo into Pagure for more transparency/visibility: https://pagure.io/fedora-pgm/pgm_scripts/blob/main/f/Change_Wrangler_Scripts. The README does a pretty decent job explaining the various scripts/tools that have been built using Claude Clode.

These tools have been built by using prompts based on the processes in the Changes Processing SOP (https://docs.fedoraproject.org/en-US/program_management/pgm_guide/sop/changes-process/) and an onboarding guide provided to me by Aoife which provided more details around each step. As the code was developed, we (Aoife and I) were conducting functional testing to ensure the scripts were behaving as intended. It is noted that I missed some things in a recent change that was processed (and a patch was put in place to stop adding sections that were not needed).

I understand enough about code to know how to describe what is needed and high-level understanding of how the changes could be implemented-- however, beyond that, I am in no position to understand code quality or best practices.

My scripts leverage Ben's PgM scripts (https://pagure.io/fedora-pgm/pgm_scripts) to move changes through the process.

My overall goal of this tooling is to reduce the toil and manual effort of copy / pasting from multiple sources. The solution has so far resulted in a substantial time savings.

One of my goals has been to "harden" the solution from a functional perspective and get it into Fedora Infrastructure.

It would be great to have someone with more expertise to review the code and help get it into a more reliable and steady state-- especially someone who has the Fedora AND coding background.

I apologize for not replying in-line, but was trying to consolidate and provide a high-level update.

What I'd like to see:

Let's use a Workflow category on discourse to propose and draft Changes. We could use the Discourse Forms feature https://meta.discourse.org/t/placeholder-forms/113533 to replace the template, and possibly https://meta.discourse.org/t/kanban-board/118164?tl=en for the workflow. Then, once the change is ready for community feedback, move it to the existing discussion category.

Once the change is accepted and finalized, it could then be transferred to the wiki so we have a preserved record. (Or to a new Docs site location.)

I already dislike interacting with Discourse for Changes, I definitely don't want to go whole hog into writing them directly into Discourse.

@ngompa Well, writing in Discourse is basically just writing in Markdown. If you hate the Discourse web UI you can copy/paste the template into a text editor, do all your writing there, then copy/paste it back. (I'd assume most people already do that with the Wiki template).

My main point is that it's nuts to have this conversion between two text markup formats (wikitext and markdown) as part of the process. Of the two, wikitext is more finicky and less well known, and we are already trying (over the long term) to deprecate the wiki.

We can talk about more radical overhauls, but I think just dropping the wiki out of the process is a fairly easy tweak to the current approach which would make the automation substantially easier and avoid problems like the one discussed in this ticket, which is why I brought it up here.

We are very likely to change to the new forge before the next election cycle. So at least pagure will be out of the equation. I think it doesn't make sense for @alking or @adamwill to spend time on improvements to code when a lot of this will need to be redone in a few months anyway.

@zbyszek off the top of my head, all the tool would really need to do with the issue tracker in either case is create a couple of issues with fairly basic attributes. I don't think implementing that with both pagure and forgejo APIs would be really a huge lift, so I think it's viable, but I didn't actually try yet.

Just poking through it I see a few obvious issues, like format_code_blocks is called in various places but is a complete no-op, and it's doing a lot of wiki text parsing which it would probably make sense to outsource to mwclient or another wiki library. More broadly, you'd want to look at just using pandoc to do to the conversion between wikitext, markdown-for-discourse and markdown-for-email - this kinda looks like it implements it all from scratch.

Heavy +1 here, pandoc can probably do most of the conversion already, and things it cannot handle can probably be automatically fixed.

Further discussion and effort is pending forge switchover.

The Bugzilla tracking bugs are also broken. The first Change owner should be the assignee but all the Change owners should be CCed.

I don't think that has anything to do with AI.

The existing scripts try to handle that - see https://pagure.io/fedora-pgm/pgm_scripts/blob/main/f/changes/createbz.py#_48 and https://pagure.io/fedora-pgm/pgm_scripts/blob/main/f/changes/createbz.py#_105 - but it's imperfect because sometimes people don't write their names/addresses in a format it can parse, and sometimes they don't use the same address they use in Bugzilla. The wrangler is supposed to check this manually.

I have created https://pagure.io/fesco/issue/3524 to discuss the general process improvements raised in this ticket.

I don't think asking FESCo for opinions is going to help much. Obviously we want the tickets to be filed nicely formatted and without corruption of the text. If the tooling needs to be fixed, then it needs to be fixed, but the details cannot be decided by the committee.

Wouldn't it be useful to take that as an opportunity to think of updating the process with the wiki, etc.?

Some conversations at CentOS connect sounded to me getting rid of the wiki (and wikitext) in the process is getting more and more likely. Discourse is md text to which most of us are used to and it is effectively human readable (unlike the ugly wikitext) as well: even if we have no final solution yet, discourse text (md) can be copy/pasted to the mailing lists in the beginning (the audience of the mailing lists definitely can read md, don't they?).

I saw proposals written by others in which the AI also did formatting changes (though I didn't see text changes in the meantime), but as long as there is an AI involved, we cannot say for sure, as each not-yet-tested context and each AI-software-update can cause unforeseen/unintended behaviors, by nature. But creating something deterministic for a process we might want to replace anyway is not efficient to invest resources in imho.

So suggestion: rather than repairing a tool to re-establish the old process, a tool that uses Discourse's API to forward+document proposals to our forge might be more useful (what I heard at CentOS Connect sounded to me that having the proposals in our own infra too was a major concern?).

This is still an issue. The updates to ticket status after Change Proposals were reassigned or rejected as part of the "f44 incomplete changes" process required numerous fixes and updates. But at this point, I think we should wait for the move to forgejo and revisit the topic during/after that.

Proposal: Close this issue for now, wait with doing Change Process improvements until after the move to forge.fp.o when it's more clear what our future tooling will be.

This was discussed during today's meeting (logs):

INFO: The three tickets related to Change Proposal handling will
be closed. We acknowledge that the way Change Proposals are currently
handled is error-prone and partially broken. We will attempt to work
around holes in the current process and / or try to address them by
working with the current Change Wrangler. Further revamping or
improving the process can happen after the move from pagure.io to the
new forge. (@decathorpe:fedora.im, 18:22:38)