Fedora-Council/tickets

The Fedora Council uses this to record ongoing work and to track issues which need a specific resolution.  |  http://fedoraproject.org/wiki/Council

#1 Periodic user/contributor survey

Created 2 years ago by mattdm
Modified 6 months ago

We have little consistent data on our user and contributor base. It would be nice to have more.

There have been several discussions of an annual user survey in the past, but for various reasons they haven't gone very far.

From a board level, this ticket isn't necessarily about the details of doing the survey, but about finding the right interested people and enabling them (and supporting their efforts as necessary so the survey can happen consistently and regularly).


This is a companion to https://fedorahosted.org/council/ticket/16. The desired goals, methodology, and basically everything else are really different between users and contributors it will be better to track them separately.

so what is the problem to solve here? what questions are we trying to answer?

Replying to [comment:3 duffy]:

so what is the problem to solve here? what questions are we trying to answer?

This is a very old request. It came about with trying to figure out who our contributors are, what they use (software), and other questions we might want to track the answers to. Robyn and I had worked up a series of questions and even met with the person that makes Red Hat's surveys. In the end, there was little interest in doing the survey in a meaningful way and the whole idea got dropped.

I believe some of the questions we were hoping to have answers to included what desktop environments contributors use, where our contributors are, other contributor traits, how satisfied they are with the Fedora Project, etc.

Okay cool - so it seems the main goal is just to understand who is using Fedora right now and how.

I would guess the problem to solve here then - we're trying to outreach to bring more new users in, so doing annual surveys of who is using Fedora could help us understand if we are reaching those new users or not? So it could be a measurement of whether or not all the changes we're making are actually effective at bringing more folks in?

Replying to [comment:5 duffy]:
Well, specifically contributors (defined as anyone who has a FAS account). It's incredibly difficult (okay, I'll say it, impossible) to gain any useful information from an ill-defined group of people.

Replying to [comment:5 duffy]:

Okay cool - so it seems the main goal is just to understand who is using Fedora right now and how.

I would guess the problem to solve here then - we're trying to outreach to bring more new users in, so doing annual surveys of who is using Fedora could help us understand if we are reaching those new users or not? So it could be a measurement of whether or not all the changes we're making are actually effective at bringing more folks in?

+1

In addition to demographics, I'd like to get some metrics around user engagement.

We can do contributor engagement fairly well, by looking at things like badges (and basically everything fedmsg).

I am unsure how this will reach in a meaningful way those users that are not collaborators. Anyway, I think that if we measure how we are doing to serve our own people will help us. Bad response means that we are likely to lose some contributors. Good responses may be not as useful as this is a health metric. If we are serving well our own people, they will be worry about other stuff.

Replying to [comment:8 yn1v]:

I am unsure how this will reach in a meaningful way those users that are not collaborators.

Without spending (possibly considerable) money, it's going to be self-selected. But we can do a number of things, including putting the survey on start.fpo, or even including it as a link we ship as a webapp in the desktop variants of the distro, as well as having ambassadors publicize it. And we have a fairly good social media reach.

Replying to [comment:9 mattdm]:

Replying to [comment:8 yn1v]:

I am unsure how this will reach in a meaningful way those users that are not collaborators.

Without spending (possibly considerable) money, it's going to be self-selected. But we can do a number of things, including putting the survey on start.fpo, or even including it as a link we ship as a webapp in the desktop variants of the distro, as well as having ambassadors publicize it. And we have a fairly good social media reach.

The only way to obtain useful results is to have an inclusive group. Generally throwing a survey out to the Internet will not result in useful results.

We can ask people if they want to sign up to be included in periodic user surveys, and then every six months or whatever select a randomized subset of those. It's not perfect, but much better than an open internet poll.

Replying to [comment:11 mattdm]:
What kind of people? Are these potentially not Fedora users? How will we know?

The biggest problem with surveys, and by extension statistics, is being able to say after what the information actually means. When we were originally discussing this we knew that the we couldn't say that we surveyed all Fedora contributors because we were actually surveying everyone with a FAS account. There's a big difference there.

No matter how we do it we'll always have a certain amount of uncertainty in the results. You'll always get people who aren't, or are no longer, a user/contributor but are just bored or vindictive. The hope is that you'll overshadow those people with people that you actually want to hear from.

The plan was to use an instance of LimeSurvey I had running at one point and sending out individualized invitations to all FAS account holders. Each person would receive a unique URL that would prevent the person from completing the survey multiple times. We could also track the percentage of FAS account holders that completed the survey so we'd know if we had statistically viable numbers.

I still have, somewhere, the questions we were to ask in the language that the survey specialist said would be best (non-leading, complete, etc).

If we want to survey contributors, what is the point? What questions do we want to answer? What problems do we want to solve?

I think something contributor-focused is a very different beast than user focused and we should probably have two tickets if we want to investigate both.

Replying to [comment:13 duffy]:

I think something contributor-focused is a very different beast than user focused

I absolutely agree.

Sounds good. Let's focus this one on users, and I've created https://fedorahosted.org/council/ticket/16 as well.

Just to keep a memory: I happened to trigger a mailing list discussion [1], summarized in [2] about what software user installs. This is not really a "user survey", but certainly related.

[1] https://lists.fedoraproject.org/pipermail/devel/2014-December/204804.html
[2] http://leamasblog.wordpress.com/2014/12/04/how-many-users-does-fedora-have/

I have several problems with driving this here (from the council). The primary one being with the advent of Products, I feel doing a top-level survey would not be very productive. Such a thing (particularly user surveys) would ideally be done per-Product, driven by their respective working groups.

Hmmmm. I can seem some of that argument, but there are also some universal things it'd be nice to follow, and perhaps some other things (demographics, for example) which would be nice to be able to compare across the products. Additionally, if we go with the idea of collecting a sample pool from which we draw randomly, doing that at the top level makes sense to me, and then lower-level surveys could periodically draw from that.

This is distinct from other types of feedback, like user interviews, which I absolutely agree should be done at the WG level.

I have no problems polling our own users with various questions, but I'm not sure it's going to prove to be massively useful. From a high level, it will tell us they use Fedora, they use (maybe) one of the DEs, they install some applications. Great. That tells us they use Fedora. Job done? Now if you get at the root of all these questions with "why do you..." then cool, that might be more useful. It's difficult to do that without having free form answers, and it tends to reduce the number of people willing to actually do the survey because it takes more time. Likely worth doing still, but call me skeptical I guess.

Now, probably even more difficult to accomplish, I would love to know why people don't use Fedora. What are we lacking, why are their needs met by $other_distro but not Fedora, etc. But to do that, we need to be asking people that aren't our users and contributors. I think the results would be more useful, but the execution of actually getting the survey to people and having them willing to respond would be more difficult.

One way to do that would be to make surveys during free software events. We have a worldwide ambassadors, so we could ask them to reach out to people they know that do not use Fedora to fill a survey ( and not "people from their family", more "people in LUG" ).

I think that'd be useful too, although maybe that should actually be yet a third ticket?

Closing this due to lack of movement. I still think it's a good idea, but this is incarnation isn't going anywhere. We can start again if there's interest at a later time.

Replying to [comment:23 mattdm]:

Closing this due to lack of movement. I still think it's a good idea, but this is incarnation isn't going anywhere. We can start again if there's interest at a later time.
For what it's worth, I think the idea behind this survey is a current objective identified by the [https://fedoraproject.org/wiki/Diversity Diversity team], and it's also a target item for our FAD to move closer to [https://fedoraproject.org/wiki/FAD_Diversity_2017#Diversity_survey deploying a survey] targeted to better understanding how our contributor community is composed. Even if this ticket is closed, I think this idea is still being worked on in other places too.

Login to comment on this ticket.