#78 Automating a check-in to evaluate activity of members
Closed: Invalid 3 years ago Opened 3 years ago by jflory7.

Problem

As mentioned in previous meetings, some people mentioned the idea of regular "check-ins" with sponsored members of the CommOps FAS group to evaluate activity, receive feedback from contributors, and more.

Analysis

This will ensure that active contributors are kept in the group and keeps an accurate number of active contributors available to the group.

Enhancement Recommendation

The Infrastructure team does a manual, email-based check-in on their mailing list for this. We could try to adopt a similar format at a monthly or quarterly basis.


@bee2502 and I had some discussion on a way to automate this without having too much strain on a single person remembering to send out a regular reminder. One way we could do this is by setting up a once-a-month Fedocal event for a check-in with the questions in the event description. People could reply to those on the CommOps mailing list (similar to how they do for Infrastructure).

Questions to answer…

  1. Who to email? commops@lists.fedoraproject.org or commops-members@fedoraproject.org?
  2. Where to reply? (can I send Fedocal emails from a "mailing list" instead of a FAS username?)

Might be a good idea to ping pingou on some of these questions (although he is out all of next week, so maybe revisiting this 2016-08-23 meeting.

Discussed in 2016-08-16 meeting.

A really late follow-up to this one… but I revisited the ideas we discussed at this meeting and I also played around with Fedocal a bit. I'll summarize the ideas from the meeting and then give my current thoughts.

Summary from 2016-08-16

  • Two ways to do it: Infra-style with a manual post sent monthly, or could use Fedocal to automate this once a month as check-in calendar event to query members on regular, consistent basis. Non-responders would be removed from the FAS group hypothetically. Need to discuss implementation and if this is something we want to do.
  • Would this be necessary for active members to also complete every month? Is it better to separate into two "groups" - apprentices and regular members?
  • Blatantly removing people is the wrong approach: it will be demotivating and isolate people from wanting to contribute
  • If any kind of consistent follow-up is to be done, it really should be automated so it will always be consistent
  • Start in a beginner group, then move to an "alumni" group for contributors who are consistent and active (i.e. they no longer have to do the regular check-in)
  • A single ping is easy to miss and it could cause someone still interested to be left out
  • Responding to one email may not be the best gauge of activity / contributions, so makes it difficult to use as a tool to evaluate merits
  • Every team could have "three big things" an active person does, follow up on them around a 3-month interval, judge inactivity on those criteria
  • Automated tools don't flag some for removal, but just review / follow-up
  • Instead of removing contributors, "retire" them - could de-provision by removing from group, but note them somewhere along with their contributions, kind of like a wall of fame sort of idea

My thoughts

I have two competing thoughts on this right now.

  • Activity check: I like this idea for having regular check-ins / feedback from newcomers via email for where they are in the process, giving any thoughts on their experiences so far, and getting their voice involved in a small way each month. I think this is a great way to encourage participation and help show who is hiding in the back row.
  • Demotivating: As @tatica mentioned in that meeting, I want to be very very careful about how we approach this. If the consequence of not replying to an email is removal from the FAS group, I would want to make sure that we are only doing this for people we are certain who have moved on… it's easy to take away something simple like a FAS group membership, and accidentally cause someone to feel rejected or demotivated that they "didn't do enough", if there is not enough communication about what's happening. I'd rather err on non-removal than removal.

But before we can discuss things like what approach to take to how we handle non-response to the email, I'm trying to figure out the best way to determine the qualification for automating this. Right now, I could send a Fedocal email monthly to either the CommOps mailing list or to the email addresses of all sponsored members of the FAS group for CommOps. This will include active contributors as well. I don't think we need to have active contributors filling out a monthly survey of questions if they don't want to. I'd like some kind of better way to filter this to newly sponsored members or even just newcomers who are interested. Are there any better ways to implement this?

I'm beginning to think the best resolution to this ticket may be a "wontfix", and we could consider sending out an occasional questionnaire / request for feedback without implications on membership.

Thoughts?

Discussed in 2016-09-13 meeting.

We covered the ticket in the meeting, but didn't really discuss it since this week was lower attendance than normal. We opted to try to get comments and feedback in the ticket before the next meeting about this. I'm curious to know how others feel about this.

Discussed in 2016-08-16 meeting.
A really late follow-up to this one… but I revisited the ideas we discussed at this meeting and I also played around with Fedocal a bit. I'll summarize the ideas from the meeting and then give my current thoughts.
Summary from 2016-08-16

Two ways to do it: Infra-style with a manual post sent monthly, or could use Fedocal to automate this once a month as check-in calendar event to query members on regular, consistent basis. Non-responders would be removed from the FAS group hypothetically. Need to discuss implementation and if this is something we want to do.

Will the contributors be given a stipulated time period to reply to the mail? A contributor may be on a vacation or may be inactive due to some personal reasons. That shouldn't mean that he/she is completely inactive and be removed from the group.

Would this be necessary for active members to also complete every month? Is it better to separate into two "groups" - apprentices and regular members?
Blatantly removing people is the wrong approach: it will be demotivating and isolate people from wanting to contribute
If any kind of consistent follow-up is to be done, it really should be automated so it will always be consistent
Start in a beginner group, then move to an "alumni" group for contributors who are consistent and active (i.e. they no longer have to do the regular check-in)
A single ping is easy to miss and it could cause someone still interested to be left out
Responding to one email may not be the best gauge of activity / contributions, so makes it difficult to use as a tool to evaluate merits

I don't think splitting into two groups will suit CommOps. There are multiple contribution pathways emerging from CommOps which can hamper our progress. Maintenance is another issue.

Every team could have "three big things" an active person does, follow up on them around a 3-month interval, judge inactivity on those criteria

At present, I'm doing just "one big thing". Does that I mean I'm not active enough and entitled for a demotion?

Automated tools don't flag some for removal, but just review / follow-up
Instead of removing contributors, "retire" them - could de-provision by removing from group, but note them somewhere along with their contributions, kind of like a wall of fame sort of idea

My thoughts
I have two competing thoughts on this right now.

Activity check: I like this idea for having regular check-ins / feedback from newcomers via email for where they are in the process, giving any thoughts on their experiences so far, and getting their voice involved in a small way each month. I think this is a great way to encourage participation and help show who is hiding in the back row.
Demotivating: As @tatica mentioned in that meeting, I want to be very very careful about how we approach this. If the consequence of not replying to an email is removal from the FAS group, I would want to make sure that we are only doing this for people we are certain who have moved on… it's easy to take away something simple like a FAS group membership, and accidentally cause someone to feel rejected or demotivated that they "didn't do enough", if there is not enough communication about what's happening. I'd rather err on non-removal than removal.

But before we can discuss things like what approach to take to how we handle non-response to the email, I'm trying to figure out the best way to determine the qualification for automating this. Right now, I could send a Fedocal email monthly to either the CommOps mailing list or to the email addresses of all sponsored members of the FAS group for CommOps. This will include active contributors as well. I don't think we need to have active contributors filling out a monthly survey of questions if they don't want to. I'd like some kind of better way to filter this to newly sponsored members or even just newcomers who are interested. Are there any better ways to implement this?
I'm beginning to think the best resolution to this ticket may be a "wontfix", and we could consider sending out an occasional questionnaire / request for feedback without implications on membership.
Thoughts?

Maybe we can write some tool which will give us some statistics about the contribution activities by the members of the group. Datagrepper can help us with that. This way, we'll not rely on manual confirmation for a mail and we'll also know how many of the group's members are really active. I'm not sure whether this is practically possible to implement but something that crossed my mind. Thoughts?

If this isn't possible then maybe we can resolve that this ticket is a "wontfix".

@dhanesh95 Realize I never replied to you here, my mistake.

Maybe we can write some tool which will give us some statistics about the contribution activities by the members of the group. Datagrepper can help us with that. This way, we'll not rely on manual confirmation for a mail and we'll also know how many of the group's members are really active. I'm not sure whether this is practically possible to implement but something that crossed my mind. Thoughts?

If this isn't possible then maybe we can resolve that this ticket is a "wontfix".

I think the best solution for this would be to develop an automatic tool that evaluates contributor activity and makes it clear when a contributor has become absent from the community, so we can attempt to do some kind of follow-up. But I think this ticket is really blocking on an automatic tool to do this. For now, I'd rather do it manually.

For that reason, I think I am going to close this ticket as wontfix until we have some sort of tool available to harness this. With the size of the CommOps group presently, I'm not that concerned with this as a problem right now.

@jflory7 changed the status to Invalid

3 years ago

Login to comment on this ticket.

Metadata