Replying to [ticket:175 kparal]:

Currently in our installation matrix there are some test results which are from "anonymous" source (no name filled in). Example:

https://fedoraproject.org/wiki/Test_Results:Fedora_15_Alpha_RC2_Install

I was little confused by that, but rhe explained me that those are the test results that were transfered from the previous test result (i.e. from F15 Alpha RC1 in this case). We already agreed if would be better if we added <ref> with explanation to each such result, so other testers understand the meaning of it (I suppose many of them might be confused same as I was). Still, I think there are some improvements that could be made. I'd like to discuss them in this ticket.

Thanks for raising this topic, it was originally discussed at:
- https://fedorahosted.org/fedora-qa/ticket/84

I supposed testers would look up the unknown results icons at [https://fedoraproject.org/wiki/QA:Fedora_15_Install_Results_Template#Key key section] where I explained. But it seems this part could be ignored, I agree to have a more visible solution.

My proposal:
1. Transferring old results to the new matrices is a great idea, I strongly support that.
2. In order to avoid confusion about "anonymous" results, we should clearly separate new results from transfered results. Using a <ref> comment is possible, but I'd like to do something more visible. We could do something like {{result|fail|rc1|12345}} and then link rc1 to correct wiki page (if {{result}} template is not suitable for it, we could create {{oldresult}} template or similar). But even though it could be scripted, it seems like too much work. Easier solution is to do {{result|fail|previous run|12345}}, and if "previous run" is linked to [[User:previous run]], we could use that wiki page for short explanation.

{{result|fail|previous rc1 run|12345}} is good for me. Why do we need a link here? The result has already been transferred to the page, I don't think every previous result should have a link to previous page.

1. We should transfer only fails and warnings, not passes. The rationale is that the pass result makes you (me, many people) think: "Hey, there's a pass, let's work on something else". But pass from previous run doesn't mean pass in current run (as I experienced today, when I changed two previous passes to fails). Therefore warnings and fails are good to know (we can check whether they are fixed or not, and we won't forget about them), but passes are counter-productive - let's leave empty fields instead.

The transfer of results depend on the change of anaconda/certain components, not the results themselves in my opinion. It's not necessary to retest many cases just because they passed in previous run. For example, if the change of a new build won't affect the partitioning and recovery parts, which were all passed in previous runs, do we need to test them again and again in every new run? Certainly I can't guarantee the passed case still pass in new run, that's why these results should be distinguished with new tested ones.

1. All transfered results must have bugzilla number assigned. Otherwise there's no information value. If I don't know what was broken, I can't check whether it's been fixed or not.

Ah yeah, when one issue affected all cases in the same area, I used to only assign a bug id to one case, use {{result|fail}} to the others. It's not due to the transfer, I will assign the bug number to all cases to reduce the confusion.

1. We should document all of this in the "Test Results Format" table and also advise people how to interact with these transfered results. My idea: If there was a transfered warn/fail, and you checked that the problem is fixed, remove it from the table (and put your result in there instead). If you checked that the problem is still present, again remove it from the table and put your result in there instead (referencing the same bug number). If you didn't check the problem, put your result in the table and leave the transfered result intact.

I can see what you mean, but I'm afraid that rule will make posting results more complicated, and also different environments could have different results. Sometimes I can't reproduce a issue even though there's not updates fixing it. If I know it's not fixed, I will keep previous result, if I don't know, I will just remove it. So my general idea is: if you test the case carefully and step by step according to the case, you can replace the previous result with yours. I assume most testers will pay attention on the previous results and carefully remove them. Besides, we have history rollback and bugzilla for tracking, do we really need such strict rule?

Replying to [comment:2 rhe]:

Thanks for raising this topic, it was originally discussed at:
- https://fedorahosted.org/fedora-qa/ticket/84

Ah, that ticket slipped my attention :-)

{{result|fail|previous rc1 run|12345}} is good for me. Why do we need a link here? The result has already been transferred to the page, I don't think every previous result should have a link to previous page.

I just supposed that [[User:previous run]] could explain a little how to result transfer works. So noone thinks "previous run" is a real user name :-) But you're right, that's not necessary. And "previous rc1 run" is even more explanatory. No need for links.

Ah yeah, when one issue affected all cases in the same area, I used to only assign a bug id to one case, use {{result|fail}} to the others. It's not due to the transfer, I will assign the bug number to all cases to reduce the confusion.

Thanks, that will help. Maybe we can share the reference number somehow (so the bug is not duplicated many times)?

The transfer of results depend on the change of anaconda/certain components, not the results themselves in my opinion. It's not necessary to retest many cases just because they passed in previous run. For example, if the change of a new build won't affect the partitioning and recovery parts, which were all passed in previous runs, do we need to test them again and again in every new run? Certainly I can't guarantee the passed case still pass in new run, that's why these results should be distinguished with new tested ones.

We can never be sure where we can find some regressions. Even comments change can break up stuff :-) OTOH I understand saving our resources. I have split opinions on this. But if you selectively transfer only "probably safe" passes, I am absolutely fine with that.

1. We should document all of this in the "Test Results Format" table and also advise people how to interact with these transfered results.
   I can see what you mean, but I'm afraid that rule will make posting results more complicated, and also different environments could have different results. Sometimes I can't reproduce a issue even though there's not updates fixing it. If I know it's not fixed, I will keep previous result, if I don't know, I will just remove it. So my general idea is: if you test the case carefully and step by step according to the case, you can replace the previous result with yours. I assume most testers will pay attention on the previous results and carefully remove them. Besides, we have history rollback and bugzilla for tracking, do we really need such strict rule?

I see. So what about this - no rules at all. Let the tester just add their results, don't recommend them any replacing of the old ones. At the end of the day we will look over it and decide where we believe the old one should be kept (or transferred to the next run) and where it should be erased. I think it's better to count on our judgement than on arbitrary tester's judgement. But I have no hard opinions in here.

Replying to [comment:3 kparal]:

Replying to [comment:2 rhe]:

Thanks for raising this topic, it was originally discussed at:
- https://fedorahosted.org/fedora-qa/ticket/84

Ah, that ticket slipped my attention :-)

{{result|fail|previous rc1 run|12345}} is good for me. Why do we need a link here? The result has already been transferred to the page, I don't think every previous result should have a link to previous page.

I just supposed that [[User:previous run]] could explain a little how to result transfer works. So noone thinks "previous run" is a real user name :-) But you're right, that's not necessary. And "previous rc1 run" is even more explanatory. No need for links.

Okay, then I'll begin using the format as {{result|status|previous <build> run}} from the next text event.

Ah yeah, when one issue affected all cases in the same area, I used to only assign a bug id to one case, use {{result|fail}} to the others. It's not due to the transfer, I will assign the bug number to all cases to reduce the confusion.

Thanks, that will help. Maybe we can share the reference number somehow (so the bug is not duplicated many times)?

Yes, how do you think the reference No.29 in [https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test#References current install test]?

The transfer of results depend on the change of anaconda/certain components, not the results themselves in my opinion. It's not necessary to retest many cases just because they passed in previous run. For example, if the change of a new build won't affect the partitioning and recovery parts, which were all passed in previous runs, do we need to test them again and again in every new run? Certainly I can't guarantee the passed case still pass in new run, that's why these results should be distinguished with new tested ones.

We can never be sure where we can find some regressions. Even comments change can break up stuff :-) OTOH I understand saving our resources. I have split opinions on this. But if you selectively transfer only "probably safe" passes, I am absolutely fine with that.

Right, transfer task need be very careful. So when all tests in a test area seem safe passes, I would still keep at least one test with no previous result so that every test area is tested in each run.

1. We should document all of this in the "Test Results Format" table and also advise people how to interact with these transfered results.
   I can see what you mean, but I'm afraid that rule will make posting results more complicated, and also different environments could have different results. Sometimes I can't reproduce a issue even though there's not updates fixing it. If I know it's not fixed, I will keep previous result, if I don't know, I will just remove it. So my general idea is: if you test the case carefully and step by step according to the case, you can replace the previous result with yours. I assume most testers will pay attention on the previous results and carefully remove them. Besides, we have history rollback and bugzilla for tracking, do we really need such strict rule?

I see. So what about this - no rules at all. Let the tester just add their results, don't recommend them any replacing of the old ones. At the end of the day we will look over it and decide where we believe the old one should be kept (or transferred to the next run) and where it should be erased. I think it's better to count on our judgement than on arbitrary tester's judgement. But I have no hard opinions in here.

Good idea. Agree with it. :)

I've modified the key section to:

• https://fedoraproject.org/wiki/QA:Fedora_15_Install_Results_Template#Key

Do we need to add more instructions? Feel free to adjust it.

Great, thanks, rhe.

A small note: Is it more correct to use expression <build> or <compose>? I don't know.

Also, we can name the result "previous rc1 run", "previous run (rc1)", "previous run", whatever seems suitable to you.

I think we can close this ticket, the transferred results are going to be much more self-explanatory now.

Replying to [comment:5 kparal]:

Great, thanks, rhe.

A small note: Is it more correct to use expression <build> or <compose>? I don't know.

I'm not sure either.:)

Also, we can name the result "previous rc1 run", "previous run (rc1)", "previous run", whatever seems suitable to you.

I think we can close this ticket, the transferred results are going to be much more self-explanatory now.

Let's close this ticket now, I'll make adjustment accordingly later in need. Thanks.