Project quality indicators

Ratings, reviews, and statistics/metrics for projects on Drupal.org to help determine their quality.


Submitted by

Related *.drupal.org issue/discussion: http://drupal.org/community-initiatives/drupalorg#metrics

Stage: In Review

Feedback Score

169 votes
Voting Disabled

Idea Details

Vote Activity (latest 20 votes)

  1. Upvoted
  2. Downvoted
  3. Upvoted
  4. Upvoted
  5. Upvoted
  6. Upvoted
  7. Upvoted
  8. Upvoted
  9. Upvoted
  10. Upvoted
  11. Upvoted
  12. Upvoted
  13. Upvoted
  14. Upvoted
  15. Upvoted
  16. Upvoted
  17. Upvoted
  18. Upvoted
  19. Upvoted
  20. Upvoted
(latest 20 votes)


  1. Status Changed from Active to In Review
  2. The idea was posted


  1. Comment

    Does this belong on the main site. Maybe make a sub-domain for it, but I don't like the idea of rankings being on the official D.O

    Comments on this comment

    1. Comment

      there is already an unofficial site at http://drupalmodules.com/ doing similar stuff. The idea here is to bring that functionality onto the project pages themselves.

  2. Comment

    I completely understand why this would be wanted especially as a site builder or someone new to drupal. I just think it would be difficult to really have anything accurate as versions are always changing and dev versions and git repo versions are always updated.So users could place a rank based on an outdated dev version, or an alpha/beta version without understanding it's in alpha/beta/dev for a reason. I would also question who would be ranking them and on what basis. For instance, what about panels and context modules? There are those who use panels solely(even PE), those who use just context, and those who use just both. Would a module be ranked based on a personal preference or site building strategy and not really on it's individual merit? Or what about those new to drupal and they install views or panels, which can have a steep learning curve, and they are expecting an out-of-the-box solution(like a distro). Would they be rating it on it's level of complexity and potential initial frustration? This could easily happen especially if they are moving from Wordpress or Joomla or something, or never worked with a CMS at all. I mean, I am not a developer, just a site builder, and still find myself constantly learning and facing new things that seem overwhelming at first. And how would ratings be monitored overall?

    I really not trying to kill the idea, there are just so many factors to consider and would not want to see great modules/themes and new ideas being rejected based on inaccurate info. It may also deter folks from joining in on projects and getting involved(even if it's just trying it out and submitting bug reports), which is what the drupal community is about.

  3. Comment
    Goutam Dey

    I would like to see it rather as a popular modules with some ratings with related or similar modules tagged with it.

  4. Comment
    Sean Bannister

    I agree with including "reviews", and "statistics/metrics" but "Ratings" have the potential to be problematic. We see this in various App Stores, developers won't release their code early because they're concerned about negative votes. This goes against the principals of open source, release early, release often.

    What the "majority" of people would be down voting on is if a module was lacking the feature they needed, which once again goes against how open source works, a developer has put in their precious time usually for free to benefit the community, yet because the developer hasn't included every use case they'd get a down vote. Instead of a down vote what the developer should be getting from the community is a patch with the new the feature.

    The other problem is projects change and a users vote could change in the future, but most wont come back and change their vote.

    I realise the idea of ratings sounds great but there's negative side effects that could damage code contributions and turn drupal.org into a popularity contest. Instead of a community of people trying to help each other.

  5. Comment

    I think rather than project quality, it should be project healthiness.

    Healthiness indicators are :

    - the number of commits

    - latest activity

    - number of sites using it

    - quality of issues and reactive reponse to a bug/issue

    Those indicators are already there, but it requires to spend few minutes (seconds) on the project page to read them, and make to make our own idea on the project quality.

    No rating system could synthetise these indicators.

    If we let people vote for a project, then it becomes subjective.

    Comments on this comment

    1. Comment
      Community Member

      Recent activity is more important than someone's rating. In lieu of ratings, I'd like to see an improved/extended "related modules" section. The current related modules tends to be rather limiting in that either shows 2-3 similar, and then some remotely related, meanwhile hiding some really good ones. Go to project/colorbox, and compare it's related projects to the link at the bottom of Colorbox 'similar modules'.

  6. Comment

    I like the idea of having all the info maybe in a block or something, just to see it at a glance, but still am very wary of any type of ratings or healthiness indicators. Again, I completely understand the motivation behind wanting it, and am in no way trying to offend anyone or dismiss all the good input given by supporters of this. I just still think that even rating or providing any overall indicator would not accurately represent a project. For instance, if you judge things by by even the number of sites using it, you are exempting/penalizing projects that are for a very specific functionality for a specific type of site, unlike views,ctools, etc.. that everyone uses; also new projects looking for exposure and testers would be penalized as well. Quality of issues and response would also put a terrible burden on maintainers, I mean what if they actually take a vacation or get sick and then their rating goes down? Or what if they are actively seeking co-maintainers because they already know they have too much on their plate, but instead of abandoning the project they are seeking help. In this instance would some make a decision to postpone or abandon a project just for ratings? And the number of commits and latest activity can be deceiving as well. There are many modules, even just basic utility modules that just do what they do and just work and do not require a massive number of commits or updates. Like the admin role module for D6 before the functionality was included in the D7 core.

    Again, I am not trying to offend, I am only concerned about the potential negative impact it could have on developers and also new users. So I totally agree that no rating system could synthesize these factors and just wanted to provide some specific instances to consider where it could be problematic.

  7. Comment

    I agree with your comments. Some modules designed to do one particular thing just work fine and need no updates, very few people post issues because it doesn't have bugs and usage is very simple.

    So I'm saying that there can't be an overall rating.

    Overall rating based on what? Like/dislike? Number of sites using the module? Number of (open/closed) issues?

    I say : use the information already there and make your own idea.

  8. Comment

    It may be worth doing ratings in conjunction with some the issue raised at


    As certain modules become the defacto standard for doing something their rating rises?

    Comments on this comment

    1. Comment

      "as certain modules become the defacto standard for for doing something their rating rises" - I think that's a great idea to think about. How and who would make that decision, I have no ideas off hand, just the question for now.

  9. Comment

    Possibly a "I use this" button next to the solution so that as people implement we can see who is using what?

    Comments on this comment

    1. Comment

      great idea:). Maybe even expand it a little with a link to more specific info, like what type of project it is and what the needs are and how this module filled this need, or just expressing that it's a personal preference. For instance the whole panels/context debate that started and now alot of people use them both. And even the folks from development seed(original developers of the context module) stated that because they primarily build applications with Drupal, so Context fit their specific needs for that type of site, but that when needed they would "pull out the big guns" and use panels.

  10. Comment

    I think a simple up/downvote system could be problematic, as other people have mentioned. But a simplified project health indicator could be pretty simple to code based on frequency of commits, number of committers, most recent commit, average response time to issues in the queue, how many issues are marked as "critical" in the queue, if there's a "is project X abandoned?" issue, how many reported sites use the module... that sort of simple indicator.

    You know, the things we pros know to look for by hand and evaluate. It won't replace that evaluation, but it's an aid and a quick reference. Plus, it could be used to sort modules on a search page to give SOME idea of the most used/stable/reliable module responding to your search terms.

    The system should be pretty vague, because you cannot get a precise idea of the health of a project in a computed way. Color coding might work, or even just flags if those indicators reach a certain threshold. So a badge could appear for "new project" if there are fewer than 10 commits, and "infrequently maintained" might appear if there have been 6 months without a commit. "Unsupported" could appear if the average wait time on an issue is more than 3 weeks.

  11. Comment

    I think the rating could be like

    Overall level of recommandation for the module

    from 1 (very easy) to 5 (very tricky) rate for

    - the installation/configuration

    - the use

    Bugs experience with the module : from "0 or very minor" to "so many bugs I gave up using the module"

    + the indication of the version of the module people have made the review for and a self evaluation of the writer of the review (from newbe in drupal to rockstar)

  12. Comment
    webchick ( Idea Submitter )

    I think we'll want to do ratings/reviews as simply as possible. Look at a model like the Apple/Android App Store that we already know people of all technical skill levels know how to use, and base the spec off that, code it, and deploy it as a first pass. Further bells and whistles could be evaluated once the basics are up and running and we figure out how people do / don't use the system in practice.

    Comments on this comment

    1. Comment

      It could be trifurcated into (1) "I Like It" (2) "I Use It," and (3) "Difficulty." It seems that the latter category would help mediate (1), although, I'm not quite sure how the difficulty rating would work into things like Panels or Views because they are fairly easy to use to create very simple features, but they are hard to use to create complex/advanced features. Also, "meta-projects" like features might be hard to rate in this schema.

  13. Comment

    Would these reviews / ratings be per version like on wordpress.org? Otherwise they would be kind of pointless. A newer version may provide a totally better experience.

  14. Comment

    I am a fan of automated metrics on the issues queue and to a lesser extent on commits as these are more resistant to gaming. While general like/dislike voting has it's place, I prefer more focused questions such as "I found this module easy to install and configure". I believe there is no such thing as the one true perfect metric, so I recommend that no-one aims for a single module score based on weightings of other scores.

  15. Comment

    I think having a review/rating style that has a set of metrics comp.. e.g.:





    Quality of Code:

    Site used on:

    Some of these could be automatic metrics, e.g. Community might be calculated by some metric using commits, issue responses, and releases. Sites used on is obvious.

    But some of them will have to be user submitted. For user input, there should be a review text area and a way to look at high/low reviews. Like most e-commerce sites do, (e.g.: www.compusa.com), so you can filter the "ax to grind" crowd out.

    The user input stuff can be done fairly easily using FiveStar content fields and views.

    Possibly, something like conditional_fields could be used to map "what you can rate" to some "user experience" fields... E.g. ask for experience with drupal and include "do code" as an option.. then only allow these folks to rate the "code quality".

  16. Comment

    It could be something very simple like 'I would recommend this module to others - yes / no'.

    Comments on this comment

    1. Comment

      A main problem with recommendation falls into the questions of complexity and general use value. For the latter: some projects are used almost universally (i.e.: ctools and views), while others have more limited use cases. So, the recommendations would be problematic in the contextual aspect of them. One nice way that this has been addressed in the past is the creation of the "module comparison" pages; that way, for instance, one can look through access control modules in order to see what they might need based on preference and scope for any particular project (i.e. should I use TAC or TAC Lite?). The scope issue might affect the "recommend" to others aspect.

      As for the complexity: some modules, no doubt, are more complex to use while others are basically plug and play. And there are others that are easy to use for simple cases but can also be used in complex ways to achieve amazing things (again, views, ctools). If a newer user tries out a more complex module and can't make heads or tails of it, then they might hit the "don't recommend" button because the implementation is above his/her head. So, those problems might skew the results negatively.

  17. Comment

    I kindly thumbs down, I think its a good idea but it might jeopardize fairness within the community, unless a very detailed metric system is created for small, big, old, and new projects.

    After reading all the comments, I wanted to change my vote. This clearly is worth discussing.

  18. Comment

    I would not go for the simpliest possible rating. For me it works for something as android market because it doesn't require any involvment. It takes 20s to install, I try it out, I don't like I uninstall in 20s also.

    But installing a module requires more.

    When I want to buy a product, it's important to me to have more info. For instance this beauty test products (sorry guys ;)) http://www.beaute-test.com/serum_eclat_extraordinaire_-_code_jeunesse_lumiere_l_oreal.php is very precious. There is first an overall rating for the product but it is based on a detail rating (on texture, smell, efficience, ratio price/quality). Plus the comments the user may leave.

    I would go for something like that for the module rating. Simple rating would be easier to put in place, but for me simple rating would be more or less useless if I don't get any details.

  19. Comment

    PS : of course installing a module is not buying something. But it requires time to get into it, and time has value.

  20. Comment

    As many mentioned, some indicators of quality exist already (like # of sites using the module) and some more could be valuable, like a rating system etc. But I think it will bring huge dividends if we had a designer make these indicators the easy to find (they ought to be the main feature of a module page IMO). It's a small investment that would go a long way. I really battled until I finally found out where the existing indicators were buried.

  21. Comment

    This is a good idea and will encourage users to register on Drupal.org too.