#29 Handling multiple user input in installation matrices
Closed: Fixed None Opened 14 years ago by kparal.

Currently it is not clear how to proceed in the installation matrices test cases, eg:[[BR]]
https://fedoraproject.org/wiki/Test_Results:Fedora_12_RC1_Install#Test_Matrix [[BR]]
when there are multiple user inputs for the same test case.

Suppose that some test case is already filled in (with some result) and I have executed it too. How should I proceed when:

  1. I have same result as person listed
    * add myself to the list?
    * do nothing?
  2. I have different result as person listed
    * rewrite the result? (both pass->fail and fail->pass?)
    * somehow split the result between me and person listed?

We should think about this and provide a few lines comment to the instructions section at the start of the page to describe how the people should behave in these cases. It may be very simple solution, but currently the information is missing at all.

We could also think about how the matrix could be modified to allow better display of multiple results of single test case. For example pass count / fail count (and people would just increase the counter)? Or some simple +/- button, is that technically possible?


Assigning to liam. I believe there is a guide for adding test results on the wiki (see https://fedoraproject.org/wiki/QA:Posting_Install_Test_Results).

Perhaps this can be updated to address Kamil's concerns?

I've been playing around with a new template ([https://publictest6.fedoraproject.org/wiki/Template:Testresult Template:testresult]) which would replace the 3 templates we use now ([https://fedoraproject.org/wiki/Template:Testresult/fail Template:testresult/fail], [https://fedoraproject.org/wiki/Template:Testresult/pass Template:testresult/pass], [https://fedoraproject.org/wiki/Template:Testresult/warn Template:testresult/warn]). I'm trying to accommodate having multiple testers. It's not perfect, but it seems to work okay. The template takes the following parameters:

{{{
{{testresult|fail|jlaska|123425|98776|...}}
^^^ ^^ ^^ ^^^
template result FAS Bug numbers
}}}

An example usage of the template is available at https://publictest6.fedoraproject.org/wiki/User:Jlaska/draft2

There is an issue where having nested templates isn't properly being rendered by mediawiki. I don't fully understand this, and am not able to work around at the moment. If someone is able to help figure out this problem, we can try this for the next round?

That's great! It is certainly good to know that a test case passed for 3 people than it "just passed" (for someone). If I need to link external bug, I can still do it in <ref>, right?

The appearance is not currently as appealing, that's true :) Unfortunately I'm not proficient with mediawiki templates. But even currently I believe it serves our needs.

I was also thinking about creating a simple webpage for providing the results also for people not able to edit wiki syntax, and then to populate our wiki test page somehow. But I am sure this was considered before?

Just for curiosity, I have been looking into Ubuntu QA. If you want to have a look:
* https://wiki.ubuntu.com/Testing/ISO/Procedures#Adding%20test%20reports - providing result
* http://iso.qa.ubuntu.com/ - displaying results
* http://testcases.qa.ubuntu.com/ - non-iso test cases (maybe we could utilize them some time)

Replying to [comment:4 kparal]:

That's great! It is certainly good to know that a test case passed for 3 people than it "just passed" (for someone). If I need to link external bug, I can still do it in <ref>, right?

Yeah, we may drop that from the template and we just require using <ref> as usual.

The appearance is not currently as appealing, that's true :) Unfortunately I'm not proficient with mediawiki templates. But even currently I believe it serves our needs.

Liam and Rhe ... what are your thoughts on modifying our wiki templates for F-13 test tracking (see previous comment https://fedorahosted.org/fedora-qa/ticket/29#comment:3)?

I was also thinking about creating a simple webpage for providing the results also for people not able to edit wiki syntax, and then to populate our wiki test page somehow. But I am sure this was considered before?

I think this might also touch on what lili and rhe are working on with mgracik (see https://fedoraproject.org/wiki/Is_anaconda_broken_proposal). The initial thought (keep in mind this project is still in the planning phase), is that, just like israwhidebroken.com, ''is anaconda broken'' would allow for user submitted results along with automated results.

Just for curiosity, I have been looking into Ubuntu QA. If you want to have a look:
* https://wiki.ubuntu.com/Testing/ISO/Procedures#Adding%20test%20reports - providing result

Liam has a similar page ... but we could make it more obvious (https://fedoraproject.org/wiki/QA:Posting_Install_Test_Results)? Of course it would adjust based on how our installation testing is tracked in the future. If you think there are improvements for that page ... let's kick it off if a new ticket?

I would see ''Is anaconda broken?'' filing this void for us.

We have something similar, but I do appreciate their simplified categorization. Perhaps something we could modify.
* https://fedoraproject.org/wiki/Category:Test_Cases
* https://fedoraproject.org/wiki/Category:Test_Plans

Thanks, I'm a big fan of keeping an eye on trends for other community projects and building on them. This might be something to kick off as a different thread on fedora-test-list perhaps. Mozilla (http://quality.mozilla.org/) and oo.org (http://qa.openoffice.org/) are two other high profile projects that are interesting references

Hi James, I've watched your example in #comment3, it's much better than the current one to serve more users. My thought is we can also divide each table into one/two/three/parts depending on the results condition of this case: If anyone have the same result(color), just add the name to the list. If different from others, divide the table into two(or three), add the result in another space. This way the results can be sorted out.But I'm not sure if it's possible in mediawiki.:>

Besides, I like the 'Adding test reports' part in
https://wiki.ubuntu.com/Testing/ISO/Procedures#Adding%20test%20reports
fill in a form and click 'submit', then the results put up automatically. For people who wants to test and submit manually, I think that's a better way.

As you said, anaconda broken would allow for user submitted results along with automated results. But I'm still thinking about how to combine manual and auto results together or just separately. How does 'is rawhide broken' achieve this? Thanks.

Replying to [comment:6 rhe]:

As you said, anaconda broken would allow for user submitted results along with automated results. But I'm still thinking about how to combine manual and auto results together or just separately. How does 'is rawhide broken' achieve this? Thanks.

This will be available in fedora infrastructure managed systems in the next few weeks. Until then, you can observe the interface at http://test1185.test.redhat.com:8080

There is an option to LOGIN using your Fedora account at the bottom. I'll need to investigate, since it doesn't seem to be working properly at the moment. But once logged in, it allows users to change the test result values.

Sorry for giving the reply so later.It's odd that I did not receive any mail about this ticket before HeRui asked me about this last Friday. When I saw this ticket, James has solved this problem,the solution is so great! Thanks for kparal to pointed this out. We will make it work on F13. I have a test on James's wiki, when I add more than 3 users,it looks not good, please see the bottom of the page,did I operate incorrectly? I add some lines to my draft, it seems good, but it's a bit complex, I don't know whether I can write it to template,please see my draft:
https://publictest6.fedoraproject.org/wiki/User:Lm/draft
I will do more investigation on it later, thanks

Replying to [comment:8 liam]:

Sorry for giving the reply so later.It's odd that I did not receive any mail about this ticket before HeRui asked me about this last Friday. When I saw this ticket, James has solved this problem,the solution is so great! Thanks for kparal to pointed this out. We will make it work on F13. I have a test on James's wiki, when I add more than 3 users,it looks not good, please see the bottom of the page,did I operate incorrectly? I add some lines to my draft, it seems good, but it's a bit complex, I don't know whether I can write it to template,please see my draft:
https://publictest6.fedoraproject.org/wiki/User:Lm/draft

I think it doesn't look so hot because of the padding and/or margin values specified in the CSS of the [https://publictest6.fedoraproject.org/wiki/Template:Testresult|testresult template]. We can always try another technique, this was just something quick that came to mind.

3 things I think we'd need to resolve before choosing this format:
1. Getting the CSS just right so the alignment looks good in the example Liam provided. I'm not an expert here, someone may want to reach out to the fedora websites team for guidance.
2. Figure out why nested templates aren't working. I suggest contacting wiki guru [https://fedoraproject.org/wiki/User:Ianweller Ian Weller] or seeking guidance from upstream mediawiki
3. Creating template documentation - much like we have with our existing templates -- https://fedoraproject.org/wiki/Template:Testresult/fail. I've adding an example that's included at https://publictest6.fedoraproject.org/wiki/Template:Testresult

I found Ianweller and asked help,she will work on it these days.

how does it look this time:

https://publictest6.fedoraproject.org/wiki/User:Lm/draft

I modified jlaska's template and added a tag to each line of result.

Replying to [comment:11 liam]:

how does it look this time:

https://publictest6.fedoraproject.org/wiki/User:Lm/draft

I modified jlaska's template and added a tag to each line of result.

I like those improvements! I wanted to remove the requirement having testers add a
tag, so I copied your changes, then modified the Template:testresult to use a

instead of a . Let me know what you think.

It's frustrating, but I can't figure out how to get the 3, 4, 5 and 6 positional parameters to properly display. Was Ian able to assist with that as well? Perhaps he might have some guidance there.

I like those improvements! I wanted to remove the requirement having testers add a
tag, so I copied your changes, then modified the Template:testresult to use a

instead of a . Let me know what you think.

It's frustrating, but I can't figure out how to get the 3, 4, 5 and 6 positional parameters to properly display. Was Ian able to assist with that as well? Perhaps he might have some guidance there.

I like your improvements very much.:) . About the display of the parameters, I don't know the complex syntax,but I am wondering how it will look like when some tester adds comments more than 4? If we use tag <ref>,no matter how many comments,it will display correctly,right? How do you think about this?

Looks good, is it possible to make pass, warn and fail displayed separately?

Replying to [comment:14 rhe]:

Looks good, is it possible to make pass, warn and fail displayed separately?

rhe: We can not control how the testers input their results, if the input order is not correct,the wiki must sort the results to display correctly,but I don't know whether wiki can do that automatically.

jlaska, I get there too,:) by put the tag
to template, and add all results to the same line.

Replying to [comment:13 liam]:

About the display of the parameters, I don't know the complex syntax,but I am wondering how it will look like when some tester adds comments more than 4? If we use tag <ref>,no matter how many comments,it will display correctly,right? How do you think about this?

As it's written, it will accept up to 4 additional comments. It can be changed for more or less. But I don't suggest we use this syntax until we can figure out how to get nested Templates to work properly using it. We can track that in a different ticket.

Let's close this out. Lili has the main changes into the fp.org/wiki now and it looks great (see https://fedoraproject.org/wiki/Template:Result).

I've opened ticket#42 to address the additional parameters

Login to comment on this ticket.

Metadata