Skip to content

Latest commit

 

History

History
589 lines (493 loc) · 21.3 KB

cEP-0020.md

File metadata and controls

589 lines (493 loc) · 21.3 KB
Raw

Newcomer metrics and Gamification

Metadata
cEP 20
Version 1.0
Title Newcomer metrics and Gamification system
Authors Shrikrishna Singh mailto:[email protected]
Status Proposed
Type Process

Abstract

This cEP describes a gamification system to automate the recognition of activities performed by people who join the coala community, and encourage them to progress towards obtaining the role of "Developer" in the community.

This cEP is a part of the GSoC project.

For the implementation of this process, this project will incorporate a gamification system which will allow us to track the progress of newcomers and assign some points to each of the activities they do. There will be different levels based on point thresholds and a newcomer will also be eligible for specific privileges after the completion of each level. There will also be different badges on the basis of performed activities by the newcomers.

So, every newcomer at coala will have to complete the processes involved in the gamification system in order to get the coala all-rounder badge and then they will be eligible to apply for the developers role at coala.

Background

Newcomer Experience

A newcomer at coala have to follow the newcomer guides and perform certain tasks mentioned there in order to be eligible to apply for the "Developers" role at coala.

Here is a permanent link of the newcomer guides at the moment.

The "Developers" role at coala means that the recipient is now a part of more trusted team, which is the organization's developers team on GitHub.

The problems:

A good way to keep an open source project alive is to get new contributors as the old ones will be busy in their own work someday and couldn't be able to contribute on the daily basis. So, the first and foremost problem is, how to attract more newcomers to contribute to the community?

But, with increasing amount of newcomers, chaos comes. And it will be hard to focus on each and every new contributor. So, there has to be a way to manage the same. Also, to keep the newcomers contributing to the community, a good way is needed to teach and help them so that the work on coala progress at a fast pace.

Also there has to be a way to get more effective feedback from the newcomers who have just started using the tools, organization is building. And a way to use those feedback to improve coala and its processes.

Why Gamification?

As coala tries to be a welcoming organization for newcomers, it gives them clear pathways and as much direct assistance as possible. However, it is already known that newcomers face many barriers while attempting to contribute to open source for the first time, and additional barriers while attempting to contribute to projects run by the coala organization.

Some barriers they face include orientation issues that can potentially demotivate newcomers from placing their first contribution. On the other hand, gamification is widely used to engage and motivate people to accomplish tasks and improve their performance.

Therefore, the goal of this project is to use gamification to orient and motivate newcomers to overcome onboarding barriers to contribute and engage with coala and its community.

Gamification Introduction

There will be four game design elements to help newcomers: Activities, Points, Badges and Levels. For each of these elements, there will be a set of rules to describe the operation of the gamified environment.

Activities

Following are the activities which could tracked in this system:

  • Create bug report
  • Create newcomer difficulty issue
  • Create external linter bear proposal issue
  • Create native bear proposal issue
  • Create generic bear proposal issue
  • Create documentation issue
  • Create low difficulty issue
  • Create medium difficulty issue
  • Create high difficulty issue
  • Create high importance issue
  • Create medium importance issue
  • Create low importance issue
  • Create trivial importance issue
  • Merge pull request that fixes a bug
  • Merge pull request for external linter bear proposal issue
  • Merge pull request for native bear proposal issue
  • Merge pull request for generic bear proposal issue
  • Merge pull request for a documentation issue
  • Merge pull request for newcomer issue
  • Merge pull request for low difficulty issue
  • Merge pull request for medium difficulty issue
  • Merge pull request for high difficulty issue
  • Merge pull request for high importance issue
  • Merge pull request for medium importance issue
  • Merge pull request for low importance issue
  • Merge pull request for trivial importance issue
  • Gitmate errors on pull request
  • CI status of the pull request
  • Review comments on pull request
  • Number of questions asked in coala/coala room
  • Number of answers given in coala/coala room
  • Number of questions asked in coala-bears room
  • Number of answers given in coala-bears room
  • Number of questions asked in coala/corobo room
  • Number of answers given in coala/corobo room
  • Number of questions asked in coala/community room
  • Number of answers given in coala/community room
  • Number of messages in coala other rooms
  • Make use of corobo on Gitter
  • Introduce .coafile in user own repository
  • Introduce .coafile in an organisation with 50 or more stars for its repositories
  • Introduce CI in the users own repository to run coala using Travis CI
  • Setup the Circle CI to the fork of coala repository
  • Setup Travis CI to the fork of coala repository
  • Setup Appveyor CI to the fork of coala repository

Points

Points are a simple tool to reward positive contributions to coala, and they provide an excellent base upon which to build a gamified design. A good way to start designing this system is to assign points to some of the tracked activities above:

  • Create Issue
    • 5 points will be deduced in any type of duplicate issues
    • 5 points for importance trivial
    • 5 points for difficulty newcomer
    • 5 points for external linter bear proposal
    • 7 points for native bear proposal
    • 8 points for difficulty low
    • 10 points for type/bug
    • 10 points for importance low
    • 10 points for generic bear proposal
    • 12 points for difficulty medium
    • 15 points for difficulty high
    • 15 points for importance medium
    • 20 points for importance high
  • Merge Pull Request
    • 5 points will be deduced if the pull request is duplicate
    • 5 points will be deduced if the pull request is closed without merge
    • If there are no Gitmate errors
      • 10 points for difficulty newcomer
      • 10 points for importance low
      • 12 points for writing external bear
      • 16 points for writing native bear
      • 20 points for bug fixes
      • 20 points for difficulty low
      • 20 points for importance medium
      • 20 points for writing generic bear
      • 30 points for importance high
      • 30 points for difficulty medium
      • 40 points for difficulty high
    • If Gitmate errors are b/w 1 to 10
      • 5 points for difficulty newcomer
      • 8 points for importance low
      • 8 points for writing external bear
      • 12 points for writing native bear
      • 15 points for bug fixes
      • 15 points for difficulty low
      • 15 points for importance medium
      • 15 points for writing generic bear
      • 20 points for difficulty medium
      • 25 points for importance high
      • 30 points for difficulty high
    • If Gitmate errors are b/w 10 and beyond
      • 5 points for importance low
      • 5 points for difficulty newcomer
      • 5 points for writing external bear
      • 7 points for writing native bear
      • 10 points for bug fixes
      • 10 points for difficulty low
      • 10 points for importance medium
      • 10 points for writing generic bear
      • 15 points for difficulty medium
      • 20 points for difficulty high
      • 20 points for importance high
  • Activities related to introducing .coafile
    • 5 points
  • Activities related to setting up CIs to the fork
    • 5 points

Note that these points allocations to the activities can be change during implementation, it is just an example to have a better understanding of this system.

This cEP relies on labels being up to date before a PR is merged. This can be achieved by developers checking labels on new issues, and maintainer checking the labels are up to date before merging, but systems to improve the accuracy of labelling of issues and PRs are beyond scope of this cEP.

In case if a newcomer creates a medium level generic bear proposal issue, he has met the criteria for the activities of:

  • Create medium difficulty issue
  • Create a generic bear proposal issue

Note that in this case points would not add up. There will be a method to check that which activity is assigned with the highest number of points and the newcomer will be awarded with those points. Also, there will be more method for scoring in the cases when assigining points based on the activity with higher points would not be good. E.g.:

Creating an issue which has importance/high(20 points) and difficulty/newcomer(5 points) labels.

Levels

To increase a user's competitive instinct, a series of levels can be implemented that confer rank as newcomers become more active. A good and straightforward way to award levels in a new gamified design is to base them on point thresholds. As users earn points, they move up an incremental series of levels. To infuse the level with meaning i.e.: a ranking within the system, the levels should be named in a way that indicates status.

But the levels wouldn't be enough motivation to go until newcomers have some special privileges after completing each level:

Note that the following levels are an example of possible levels with suggested privileges based on the example points allocations in the previous section.

  • Level-I:
    • Name: Fresher
    • Points_required = 0
    • Privileges:
      • No privileges for freshers
  • Level-II:
    • Name: Newbie
    • Points_required = 10
    • Privileges:
      • Get assigned to newcomers issue
  • Level-III:
    • Name: Beginner
    • Points_required = 20
    • Privileges:
      • Get assigned to a difficulty low issue
  • Level-IV:
    • Name: Intermediate
    • Points_required = 50
    • Privileges:
      • Mark your own PR as wip
  • Level-V:
    • Name: Cool
    • Points_required = 100
    • Privileges:
      • Mark a PR by another newcomer as wip
      • Get assigned to difficulty medium issue
  • Level-VI:
    • Name: Expert
    • Points_required = 200
    • Privileges
      • Assign newcomers to difficulty newcomer issue
      • Assign newcomers to difficulty low issue
  • Level-VII:
    • Name: Legend
    • Points_required = 350
    • Privileges
      • Invite newcomers to org
      • Assign newcomers to difficulty medium issue
      • Permission to ack and unack on the pull requests

Implementation of privileges is beyond scope of the gamification system. The gamification system will merely expose a list of privileges that each person has, and other systems are expected to adapt to use these privileges. E.g.: corobo

While implementing privileges, a first manual check should be done for testing purpose, and then automated only if there is no serious risk or there is a good risk reaction mechanism in place.

Badges

Another way of explicitly nudging a newcomer to action is to award badges for completing tasks. Badges enable newcomers to follow their performance and compare to other newcomers. Badges will be awarded after performing certain activities by the newcomers:

  • The bug finder:
    • Details: The one who find bugs in the existing codebase
    • Activities:
      • Create at least one issue which labelled as bug
  • The bear hunter:
    • Details: The one who create issues about new bears
    • Activities:
      • Create at least one native bear proposal issue
      • Create at least one generic bear proposal issue
  • The bear writer:
    • Details: The one who write new bears
    • Activities:
      • Merge at least one external linter bear proposal pull request
      • Merge at least one native bear proposal pull request
  • The helper:
    • Details: The one who help other newcomers
    • Activities:
      • At least 20 answers given on coala Gitter rooms
      • Reviewed at least 1 high importance pull request
      • Merge at least 1 high importance pull request
  • The reviewer:
    • Details: The one who review others' pull requests
    • Activities:
      • Review at least 3 difficulty newcomer pull requests
      • Review at least 2 difficulty low pull requests
  • The super reviewer:
    • Details: The one who is the legend of reviewing
    • Activities:
      • Review at least 10 newcomer difficulty pull requests
      • Review at least 7 low difficulty pull requests
      • Review at least 3 medium difficulty pull requests
  • The coala all-rounder:
    • Details: Has met the prerequisites to apply for the coala developer role
    • Activities:
      • Introduce .coafile in user own repository
      • Merge a difficulty/newcomer Pull Request
      • Review at least a difficulty/newcomer Pull Request
      • Merge a difficulty/low Pull Request
      • Review at least a difficulty/low or higher Pull Request

So, to be eligible to be a developer at coala, a newcomer must complete all the activities mentioned in the coala all-rounder badge. But in case if some of the newcomers are not able to be eligible to apply for coala developer role, they could take away the other badges.

Names and metrics of tracked activities will also be shown on the leaderboard along with the number of times they have been done. That will help maintainers to varify if an automatically awarded badge is correct or not.

Note that in case of an issue or a pull request has two labels which are counted in this system, the performed activities by the user will be two, and both the activities will be counted while awarding badges. So in case if there is a badge for creating a medium difficulty issue and there is a badge for creating a bear proposal issue, both badges will be awarded to the newcomer who has created a medium difficulty bear proposal issue.

Implementation

Getting the data

GitHub/GitLab

For getting the data related to newcomers activity on GitHub/GitLab to the community, this system will mainly use IGitt.

Gitter

For getting the stats of newcomers' messages, this system will use Gitter API and import all the messages by newcomers in webservices. And then those messages will be imported to the community repo into two Django models, i.e.: Question and Answer. The Question model will be responsible for storing all the questions asked by the newcomers in coala Gitter rooms. And the Answer model will be responsible for storing all the answers given by the newcomers in coala Gitter rooms. And these numbers will be shown on the user's dashboard along with each completed milestones of the newcomer. This will help us to figure out how much a newcomer has been noisy, or how much they have asked questions and how much they have helped others while completing certain tasks and not promote them to developers team until they have learned to learn things by themselves.

E.g.: Here is the log of some frequent questions asked by a newcomer while solving an issue, which clearly shows that the newcomer is noisy and has not been through newcomer docs. So, this system will count the number of questions asked by that newcomer, and show it to the leaderboard along with the issue they have completed.

A System for Tracking Activities, Points and Levels

Now that the rules for our gamification design are ready and the relevant data is imported in the community repo, a good

way to start the implementation of gamification system is to design some models and methods to track the user's activity, points and levels:

Creating the activity model

An Activity model will give a way to keep a log of each time the user earns or loses points so that there will be a record of his points history on a dashboard.

A mockup of a basic Activity Django model would look something like:

class Actvity(models.Model):
    name = models.CharField(max_length=500)
    details = models.TextField()
    points = models.IntegerField()
    number_of_times = model.IntegerField(default=0)

number_of_times field store the value of number of times an activity has been done by the same user.

The User model will have a ManyToMany relationship with the Activity model with a method named count_similar_activity which will check if the action by the user has been done previously? If Yes, it will increase number_of_times field with one.

Creating the Level Model

A mockup of basic Level Django model would look something like:

class Level(models.Model):
    name = models.CharField()
    points_required = model.IntegerField()
    user = models.ManytoManyField(User)
    privileges = models.ForeignKey(Privilege)


class Privilege(models.Model):
    name = models.CharField(max_length=500)

And a query to create a level would be:

privilege = Privilege.objects.create(
    name='Get assigned to a newcomer issue',
    )
level1 = Level.obejects.create(
    name='Newbie',
    points_required=5,
    privileges=privilege,
    )

Extending the User Model to add points and Levels

Whenever a user perform an activity his points will be increased. So, there should be some methods to check if the user has achieved a new level:

class User(models.Model):
    ...
    ...
    points = models.IntegerField()
    level = models.CharField()
    ...

    def add_points()
        a_method_to_update_points_and_level()
        a_method_for_the_log_of_the_activities()

Creating the Badges model

class Badge(models.Model):
    name = models.CharField()
    description = model.TextField()

The Badge model much like our Level model, defines a range of possible awards that a user can receive. Unlike levels, users can have many badges at one time. This sets up a many-to-many relationship between badges and users, so there is a need of an intermediate model to track which badges a user has earned. The model will be called Achievement:

class Achievement(models.Model):
    user_id = ManyToManyField(User)
    bage_id = ManyToManyField(Badge)

Now, to make awarding badges easier, let’s also add a convenience method to the User model:

def award_badge(name):
    badge = Badge.objects.filter(name=name)
    badges.award(name)

Deployment and The leaderboard

There will be Django management commands for each of the needed processes which will be triggered on each build of netlify and show the useful updated user metrics and stats on the leaderboard.

The leaderboard will have the following columns:

| Username | Activities Performed | Points | Levels | Badges |

Username column will show the usernames of each newcomer.

Activities Performed column will show all the activities perforemd by the newcomer along with the number of questions aksed and answers given on coala Gitter channels while performing those activities.

Points column will show the total points earned by the newcomer.

Levels column will show the current level of the newcomer with enabled privileges.

Badges column will show the badges earned by the user based on their activity.

The leaderboard will be sorted based on the total points earned by newcomers, which means that the newcomer with highest points will be on top and the points are directly connected to the levels, So, the newcomer at the top will also be on the highest level.

Conclusion

This project addresses the most of the problems a newcomer at coala face during the newcomer to developer process and tries to provide a simplified and automated solution:

  • A maintainer doesn't have to check manually if the newcomer has done all the activities involved in newcomer->developer process, they can just look at the points and badges earned by the newcomer through the gamification system. However, this system doesn't yet provide maintainers with the assurance of the quality of the work, so some manual verification will still be required.

  • This project provides a fun way to work in the community, newcomers will be motivated to work when they will earn points and badges even for the tiniest bit of their contributions.

  • Having a gamification system will be a good resource to attract more newcomers and the automation process could easily deal with an increase in the number of new contributors.

  • In case if some of the newcomers are not able to complete the newcomer process they can get other available badges based on their activities in the community, which will be a kind of symbol of their talent and work experience in OSS projects.

  • Keeping track of newcomers' messages on Gitter channels will help us to know how much noisy a newcomer is during performing certain tasks and not promote them to developers until they have started to find the solution of their problems on their own.