#152 Consider alternative test case management systems
Closed: Wontfix a month ago by adamwill. Opened 13 years ago by rhe.

= problem =

We've been using mediawiki to manage tests and record results, though it's easy to approach, it has limitations such as managing, tracking and querying cases+results. So is it time for us to consider another solution such as TCMS?

= analysis =

Here I quoted the suggestion from Victer Chen:

some advantages of TCMS:
* Better to record, track and query test results,
* Allow user focus on test contents instead of document maintenance.
* Share test cases

However, it couldn't be flexible as wiki. I think the most important things need balance are below:

  • Barriers
    Fedora should provide very low barriers. TCMS could be configured to allow anonymous user login and limit it's permission. Also, we can consider using fedora account to login TCMS.

  • Safety
    Wiki provides the ability to rollback content easily, while TCMS couldn't. What TCMS can do is recording all the history and allow user recovery it manually. But if we configure the permission well, it should not be a big problem.

= enhancement recommendation =

Work with TCMS team and package it to Fedora. Set up the system and use it in a small scale first. If it turns out good, enlarger the scale.


I know James has been talking about this with some RH people. We discussed it on the phone and agreed that we like the idea in principle but the ease and flexibility of the current wiki system is something we don't want to lose, so we thought we should set some quite high barriers: any tcms system must be as easy to use (both for those creating the test cases and those running them) as the wiki is.

I love the idea of doing a test event with a pilot instance of nitrate/TCMS. It was fun doing that with the semantic-mediawiki, and I think we learn a lot with that approach. We've all talked about TCMS and how best to integrate it with our workflow. There are definite shortcomings to our current wiki-based approach that limit our ability to growth, but there are plenty of positive lessons learned that I think we should recognize and prioritize.

Hurry: Would you be willing to start drafting some wiki content so we can capture this? Either a requirements or technology comparison document? I don't want this to be a free-for-all 'I want a pony' list of features we want of a TCMS solution. But perhaps starting with documenting the pro's and con's with our current wiki-based workflow? The goal for me would be to identify what our MUST-HAVE and NICE-TO-HAVE's are from any solution. Ideally, we can use that to determine the best route forward (whether it's filing bugs/tickets/rfe's against TCMS, or something else).

I've drafted [https://fedoraproject.org/wiki/Rhe/tcms_requirements_proposal a page] for your reference. So far I listed the use cases, pros and cons in wiki-based tcms and nitrate tcms, separately. Feel free to comment/discuss them. Thanks!

TCMS is really good idea for Test Creation and Manage Test plan than wiki.

Is it necessary to have permission other than view for anonymous user, like Translation team allow only FAS account user to translation on http://translate.fedoraproject.org?

Replying to [comment:4 aalam]:

TCMS is really good idea for Test Creation and Manage Test plan than wiki.

Is it necessary to have permission other than view for anonymous user, like Translation team allow only FAS account user to translation on http://translate.fedoraproject.org?

I personally consider it as a NICE-TO-HAVE, since it's one big advantage for wiki-based system to allow anonymous user editing pages with certain name space, like 'test day:' and 'Test_Results:'. So far the nitrate system only has read permission for anonymous user, but this can be customized if we review the requirement and decide to use it.

Hi,
I have found following table:[https://fedoraproject.org/wiki/User:Johannbg/Draft/QA/Test_Days/Features/EXT4].
In my opinion it can serve as draft of functionality we would like to have or we should have. Obviously i would add to the table additional column and introduced a rule that each test case has to have corresponding name like : Netoworking 1, Networking 2 and so on...

Now, i think the question is if we can do it via Wiki or TCMS.

Replying to [comment:6 xcieja]:

Hi,
I have found following table:[https://fedoraproject.org/wiki/User:Johannbg/Draft/QA/Test_Days/Features/EXT4].
In my opinion it can serve as draft of functionality we would like to have or we should have. Obviously i would add to the table additional column and introduced a rule that each test case has to have corresponding name like : Netoworking 1, Networking 2 and so on...

Now, i think the question is if we can do it via Wiki or TCMS.

Do you mean [https://fedoraproject.org/wiki/User_talk:Johannbg/Draft/QA/Test_Days/Features/EXT4 this example]? AFAIK, for wiki, one have to manually rename the page link when needed. For Nitrate TCMS, you can use 'alias' and 'tag' attitude of each test case to correspond them.

The [https://fedoraproject.org/wiki/Rhe/tcms_requirements_proposal proposal page] has been updated and written in the table format.

To further identify this issue, the following items will be analysed in details one by one:

  • Identify our current test workflows (or use cases)
  • Identify problems with our current system and workflows - e.g.
    what would we like to do but are difficult/impossible with wiki
  • Objectively compare feature sets of both tools in relation to
    the important workflows
  • Identify+prioritize any feature gaps we'd need from nitrate in
    order to make a smooth transition
  • Document a plan/roadmap for the transition

Where each item will be documented in a separate page, and the use cases page for the 1st one has been drafted:

Feel free to add comments or tell some uncovered use cases. Thanks.

Replying to [comment:6 xcieja]:

Hi,
I have found following table:[https://fedoraproject.org/wiki/User:Johannbg/Draft/QA/Test_Days/Features/EXT4].
In my opinion it can serve as draft of functionality we would like to have or we should have. Obviously i would add to the table additional column and introduced a rule that each test case has to have corresponding name like : Netoworking 1, Networking 2 and so on...

Now, i think the question is if we can do it via Wiki or TCMS.

Of course we can do it via the Wiki, as that example is done using the Wiki =) we already use a similar table format for test case results for most test days, and the validation testing. The drawbacks of using the wiki to track results are mostly in cleanliness (it's too free-form so users can mess up the tables), consistency (we don't really store all the results for any given test case in one place, instead we store results for some particular event which referenced those test cases), flexibility (see what rhe wrote) and comprehensiveness (a more sophisticated system could probably include more detail for each result, perhaps automatically retrieved from the user).

I further updated the use cases page:

And based on these, the use cases between the two tcms are compared in the link below:

Feel free to add comments. Thanks!

Hi,
I have few questions regarding to red fields by Wiki side:
1) '''Review status - manually add reviewer''' -does it mean that if create test case and perform it, thereafter i want to allow others to see the result i need to add particulars users to "list"" in order to give them rigths to see the status ???

2) '''Result summary/report - "-"''' -is it not possible at all to generate the report ???

Replying to [comment:11 xcieja]:

Hi,
I have few questions regarding to red fields by Wiki side:
1) '''Review status - manually add reviewer''' -does it mean that if create test case and perform it, thereafter i want to allow others to see the result i need to add particulars users to "list"" in order to give them rigths to see the status ???

No, even the anonymous user can view the test results on wiki. Here I mean that when create a new test plan(or test case) and wait for someone's review, there's no status as proposed or reviewed, one has to add a 'draft' sign in the page, and after review, the 'draft' sign is removed and the reviewer's name is added manually. But in Nitrate, the status of a new test plan is set as 'proposed', and after reviewed, its status will be changed to 'confirmed' AFAIK. It's easier for tracking and managing test plans/cases.

2) '''Result summary/report - "-"''' -is it not possible at all to generate the report ???
So far, we use the [http://fedoraproject.org/wiki/QA/SOP_Release_Validation_Test_Event#Report_and_Summary curl command] to extract bug report. I don't think wiki can generate report unless plugins are added.

After familiarize myself with above comparison, better fits to my own preferences Wiki.
It seems there are more useful features, sometimes not availible directly but via plugins, even though it looks more user friendly.

My score: 1:0 for Wiki

Replying to [comment:13 xcieja]:

After familiarize myself with above comparison, better fits to my own preferences Wiki.
It seems there are more useful features, sometimes not availible directly but via plugins, even though it looks more user friendly.

My score: 1:0 for Wiki

Hehe, thanks for voting for wiki. Note that the comparison is based on the use cases on wiki, so most of the compared features so far are already contained in the wiki and that's why the green fields in the wiki side are more than the nitrate side. But with deeper research on Nitrate TCMS, I'll find more features/advantages in it and add more features for comparison. Also, the red fields in Nitrate side are not permanent, they can be resolved by the tcms team if we decide those features as MUST-HAVE.

[https://fedoraproject.org/wiki/Rhe/tcms_Comparison wiki vs nitrate page] is updated again. Last time there were some unknown fields in nitrate side. By doing some tests on it, I verified several of them and added some new features. Please refer to it. So far, I think the general use cases have been covered and converted to related features for comparison.

Replying to [comment:8 rhe]:

The [https://fedoraproject.org/wiki/Rhe/tcms_requirements_proposal proposal page] has been updated and written in the table format.

To further identify this issue, the following items will be analysed in details one by one:

  • Identify our current test workflows (or use cases)
  • Objectively compare feature sets of both tools in relation to
    the important workflows

Since I called for the final review of these two item last week, from this week, I will move to next step and the pages are moved to formal links:
* https://fedoraproject.org/wiki/Tcms_use_cases
* https://fedoraproject.org/wiki/Tcms_Comparison

But feedback is still welcomed to improve them.

  • Identify problems with our current system and workflows - e.g.
    what would we like to do but are difficult/impossible with wiki
  • Identify+prioritize any feature gaps we'd need from nitrate in
    order to make a smooth transition

These two issues can be considered together. This week I will focus on identifying Must-Have and Nice-to-Have features in the [https://fedoraproject.org/wiki/Tcms_Comparison comparison table], and then summarize in another page.

Meanwhile, I will start the transition preparation work to write scripts for conversing xml files exported from wiki to files whose format can be imported to nitrate. Feel free to add comments/ideas/suggestions. Thanks.

rhe continued to work on this - the use cases and comparison pages linked above are stuffed with info and very useful - but we did not come to a final decision.

I'm going to leave this open as it's still a topic worth considering, but it doesn't need to be tied to a specific release.

...and we're still considering it. :)

Nitrate/TCMS is still a contender, the other major contender we have ATM is Moztrap: https://github.com/mozilla/moztrap . Another contender is Ubuntu's QATracker, which is more similar in design to the wiki system (I suspect they actually started off by more or less cloning our wiki design...), but has some significant limitations (it doesn't do environments, so they make each arch a separate 'build', the API is limited, and tflink isn't a big fan of the architecture in general).

In the meantime, I made some significant enhancements to the wiki system, which I call Wikitcms:

https://fedoraproject.org/wiki/Wikitcms

and built python-wikitcms and relval on top of it: https://www.happyassassin.net/wikitcms/

which address some of the deficiencies of the wiki system. We do really need to look at making Moztrap or Nitrate/TCMS compatible with our requirements, though.

....and three years later, we're still here...

python-wikitcms and relval are still there, enabling the overall Wikitcms system. The whole thing is an abomination and I'm a terrible person, but it still basically...works.

  • Moztrap hasn't had a commit since December 2015, so we can pretty much forget about that one.
  • Nitrate is still around and seems to be fairly actively maintained; was recently ported to Python 3; its requirements seem fairly fresh too (in fact a bit too fresh for Fedora ATM, but that can be fixed, or container-ized, or...handwave handwave).
  • QATracker is also still a thing, appears to be periodically updated, still has the drawbacks it had before (you need a zillion 'products' to cover everything), and is apparently a Drupal plugin.

And of course, we have our own "let's just write a new one from scratch, how hard could that be?" proposal, relval-ng.

Sooo....yeah...still a thing. And rhe isn't going to do anything about it, so clearing the assignee field. will...not...assign...to...self...will...not...assign...to...self...

Metadata Update from @adamwill:
- Assignee reset

6 years ago

Another option is Kiwi, a fork of Nitrate that is primarily authored/maintained by @atodorov , who would be keen to help us work with it if we pick it.

five years later update: wikitcms still works. we've automated a hell of a lot of the test suite anyway (82%, at last check). Kiwi is still around but nobody is in a rush to migrate to it, I don't think. we have a whole pile of more urgent things to do. let's call this wontfix.

Metadata Update from @adamwill:
- Issue close_status updated to: Wontfix
- Issue status updated to: Closed (was: Open)

a month ago

Login to comment on this ticket.

Metadata