Open Closed Merged Merge Rate 95% Confidence Interval
Women 8,216 21,890 111,011 78.6% [78.45%, 78.87%]
Men 150,248 591,785 2,181,517 74.6% [74.56%, 74.67%]
> The hypothesis is not only false, but it is in the opposite direction than expected; women tend to have their pull requests accepted at a higher rate than men! This difference is statistically significant (p < .001).
Many hypothesis are discussed in an attempt to explain this and the paper is well worth the read. Thanks to a documented GitHub API this stuff gets quite scientifically correct - everything is backed up with real-world data.
More quotes:
> To summarize this paper’s observations:
> 1. Women are more likely to have pull requests accepted than men.
> 2. Women continue to have high acceptance rates as they gain experience.
> 3. Women’s pull requests are less likely to serve an immediate project need.
> 4. Women’s changes are larger.
> 5. Women’s acceptance rates are higher across programming languages.
> 6. Women have lower acceptance rates as outsiders when they are identifiable as women.
"For outsiders, we see evidence for gender bias: women’s acceptance rates are 71.8% when they use gender neutral profiles, but drop to 62.5% when their gender is identifiable."
And then you discover in Questions Section;
"Our analysis (not in this paper -- we've cut a lot out to keep it crisp) shows that women are harder on other women than they are on men. Men are harder on other men than they are on women."
https://peerj.com/questions/2002-do-you-have-data-on-the-gen...
Good grief. You'd think, considering the near constant criticism of men in technology for gender bias, that fact might have warranted a mention as an aside at least.
One bias I don't see often mentioned in research papers is for researchers (or those who present the work of researchers) to draw light to findings supporting certain narratives while leaving findings supporting another narrative only available to the very small percentage of people who go and read the full report.
Sampling bias, perhaps? Or how that thing is called? If there's, say, 20% of women in a field and 80% men, and you get treated badly twice as likely by an average woman than by an average man, you'd still observe (in absolute terms) more instances of bad treatment from your male peers. Surely this could distort one's perception.
Most of the even-half-informed criticism I see is about systems for gender bias, and specifically acknowledges that people regardless of gender who have been brought up in a world with biases have internalized those biases.
As a man in technology, I can't say I've ever felt personally attacked for gender bias.
I guess it depends on how you look at it. When I see phrases like "boy's club" (not in this paper), it implies to me that there's a group of men who work (consciously or unconsciously) to keep women out. I'm aware that it's not at all an uncommon finding that women are pretty hard on other women, yet I think that receives very little play in the popular media.
Perhaps I do take it too personally - I do my very best to be fair and reasonable with all my fellows, whatever their various deviations (or not) from the industry norm might be. It's upsetting to me to be tarred with this guilt-by-association. I find it particularly disheartening in the venue of OSS: I've always viewed it as a truly fantastic collective charitable effort, yet lately it seems to be getting depicted more and more as some kind of refuge for white men to exclude everyone else.
I feel the same guilt, I'm a student - I'm excluded from scholarships, meetups and career events because of my gender, that plus the regular criticism aimed at the industry - I feel ashamed every time I'm asked to fill out an equality form. I don't know what to do about it.
Don't let it effect you? Or, "I look to a day when people will not be judged by the color of their skin, but by the content of their character" can be applied to sex and any other identifiable trait as well.
I think we have a winner. Most likely difference - corporate vs non-corporate contributors.
The paper's conclusion that "women are more competent overall" doesn't seem very plausible to me. If there was really a significant, consistent boost simply from being female then we'd see software companies that consist mostly of women or which are run by women outcompete opposite companies at least some of the time, and it'd be an effect that'd have been detected before. I'm struggling to think of any examples of this.
Based on my own experience of github and open source, however, the only times I have ever seen women contribute to open source, they were always being paid to do so. I cannot recall a single time that I came across a pull request or even just a regular patch from a hobbyist woman. Not sure why that is, but it has been a constant throughout my career - men do hobby programming and women don't.
Given this difference I'd expect contributions from women to be larger overall, tackling grunge work that volunteers wouldn't want to do, solves some real business problem, and more frequently be code that was tested for a while internally first so is less likely to have quality problems.
The fact that men are also less likely to have pull reqs accepted when their gender is known also seems likely to be a correlation with submitting from personal gmail accounts vs corporate accounts.
> Based on my own experience of github and open source, however, the only times I have ever seen women contribute to open source, they were always being paid to do so. I cannot recall a single time that I came across a pull request or even just a regular patch from a hobbyist woman. Not sure why that is, but it has been a constant throughout my career - men do hobby programming and women don't.
Possibly related anecdata: from my not insignificant, but still far from comprehensive experience nearly all the women who are developers in my area (not SF—I know nothing about that market aside from what I read) are with bigger companies (not necessarily software companies), and in fact there are quite a few women working as developers with those companies.
OTOH, startups and small software businesses around here do hire lots of women—but mostly as graphic designers, or in sales/marketing, or as project managers. Very, very few women even interview for startup software development jobs here (for whatever reason—I don't claim to have an explanation).
Presumably something risk related? Men take more risks, it's documented, testosterone related. Working for a startup is risky.
At Google there was (when I worked there) perhaps about 20-30% female engineering staff. Or maybe it was a bit lower, I don't recall, but it was pretty high. Yet as far as I recall, the new projects were typically started by men, I mean the bottoms-up ones. I never noticed it at the time, but thinking back, I'm struggling to think of cases where a woman started a new project with her 20% time (or even took it, actually). That might be just lack of data points of course. Projects created by employees weren't at all rare there, but it still wasn't something that happened every day.
Many women in software, by my experience, are from slightly poorer parts of the world: eastern Europe, Russia, China, India. I suspect to them programming is just a job; a way to earn good money. But it's not something they're especially passionate about in the single minded obsessively focussed way that men can be (and I'm guilty of that).
Clearly larger companies are likely to attract more risk averse candidates but they also typically offer better health benefits, better work/life balance, and are more likely to have an hr department that can maybe mitigate some of the worst sexism that crops up in our industry.
There's also probably a bit of a gravity affect happening. My entire career I've never been on a team with even a single other woman and it would certainly be a nice thing to go somewhere where that wasn't the case, so maybe it's just easier for companies that employee women to hire more women.
This is also something I have observed in my job. I am in a typical enterprise software shop and I feel like women are more likely to see programming as "just a job", while men are more likely to put more passion into it. But most people in general see it as "just a job", so maybe hobbyist female programmers are just rare because female programmers in general are rare.
I'm a woman, I program professionally and as a hobby, I have friends who do too. I think maybe the biggest problem is visibility. I work on things and have a lot of code on github but I don't really get involved with open source projects in my free time.
For me it's a combination of not wanting to deal with the very abrasive personalities idolized in some open source communities and just preferring to spend my free time hacking on throw away code to experiment with new ideas. I write clean readable tested production code at work, I want my hobby programming time to focus on impractical educational exploratory programming.
It also just feels like I have a lot less time to write code than my coworkers, even though I'm the only person on my team without kids. Traditional gender roles are evolving but it still feels like I'm left with more life stuff to deal with.
It's worth noting that the number of pull requests by women is far, far smaller than that by men. So either there are far less women active on github, or women are more careful when submitting pull requests.
The reasons for being more careful could be all over the place, ranging from actually being better programmers to being more likely to receive criticism for a bad pull request.
In an industry that's somewhat hostile to women, it wouldn't be surprising that only the most competent women stick around, whereas less competent men stick around more than less competent women, dragging average competency of men in the industry down, even if the total number of competent men might be higher than the total number of competent women.
I'm saying this in a comment that explains this point. The world is bigger than this one study.
I'm explaining how hostility to women can result in a higher acceptance rate of their pull requests. As you could have seen from the numbers in this study, women do far, far fewer pull requests than men. Why? Perhaps, because only the best women remain, and they only submit their best pull requests. That drives the average for women up, while driving the total number down.
And I don't think the acceptance rate of pull requests is a likely vector of discrimination anyway; they're fairly anonymous and focus on the code. But the hostility absolutely does exist in other areas: small talk, jokes, hiring practices, sexual harassment, etc.
I don't think the drop is the important bit - men's AR also drops when gender identifiable.
In fact, they both drop when identified to a point which (as far as I can tell from the graph; figures aren't given) they are within each other's confidence interval.
So actually, we're more critical of everyone when their gender is disclosed; equally so.
Which I find surprising. I'd have thought if anything, we'd be more trusting and supportive of the person disclosing a picture, name, etc. (the criteria used for gender identification).
I'd have thought if anything, we'd be more trusting and supportive of the person disclosing a picture, name, etc. (the criteria used for gender identification).
Why though? My experience is that aside from everyone's monkeysphere, people generally tend to hate other people.
Social media where you interact with a higher than Dunbar number of people (ie Not Facebook) is often just a nasty status game.
Humanizing a pull request can bring all sorts of baggage from that into the evaluation of a contained block of ideas.
I'm personally somewhat biased (probably unfairly) against people that use a full name and photo rather than a handle, if I don't know them. I just have this gut feeling that their pull request is probably "I've rewritten this cli tool in node.js!" or something. It's a purely tribal thing, but that may explain it to some extent (then again, I feel like github is the home of people like that, so it should work as an in-group marker? Dunno)
The same conclusion for men, using the same data set (figure 5) is this: Men's acceptance rates are 69%, but drop to 64% when the gender is identifiable.
So to summarize the two: The overall acceptance rate drops from ~70% to ~63.5% when the gender is identifyable. Very interesting.
Yeah, sorry. I was editing that comment as I initially skimmed it. I added in their final summaries and they do include that fact which you mentioned.
I'm not proud of that statistic. I used to be firmly on the fence regarding the subject of women being discriminated against in tech. With facts like that, there's no denying it whatsoever. We have a big problem.
Ignoring the fact that this difference could easily be explained away by some other factor when it is considered. It may have nothing to do with it. Causal relationships are so tricky when dealing with statistics.
PR closes can be for a variety of factors besides programmer competence or gender bias. For example, point #4 says that "women's changes are larger", whereas many large projects are particularly wary of large PRs with multiple changes. That's not to say that the large PR would break something, but many times I've seen a PR closed with a request to break it down into multiple smaller PRs that are more easily reviewed and integrated.
> 6. Women have lower acceptance rates as outsiders when they are identifiable as women.
They should have added this:
7. Men have lower acceptance rates as outsiders when they are identifiable as men.
Seeing those two together - equally justified by the text - suggests it might just be a small bias (~10%) against "outsiders" who use their real name in favor of "outsiders" who use hacker aliases...regardless of gender.
Be careful implying causation here. We would want to investigate who identifies themselves as any sex vs who remains unidentified (or less obviously identified) and the skill levels represented in each group. Since GitHub does not request your gender for your profile, they used Google+ profiles, which I think would significantly slew the results; they did not sample from all pull requests, but from those that were linked to a G+ account AND whose owners decided to post their gender.
> Specifically, we extract users’ email addresses from GHTorrent, look up that email address on the Google+ social network, then, if that user has a profile, extract gender information from these users’ profiles. Out of 4,037,953 GitHub user profiles with email addresses, we were able to identify 1,426,121 (35.3%) of them as men or women through their public Google+ profiles.
> This is awful - I think it would be good to have some discussion around what we can do as a community to improve this?
I don't agree that the result is as strong as "awful" indicate because the paper concludes that
> Women had higher acceptance of pull request than men
> Identifiable women had higher acceptance of pull request than men
> Insider identifiable women had higher acceptance of pull request than men
> Both outsider identifiable women and outsider identifiable men had lower acceptance rates than their insider counterparts
The study proposes the existence of gender bias because, in this last point, the drop for women was higher than the drop for men, regardless of all the other instances where the opposite was true.
I don't believe the study is representative for the whole industry at all (GitHub being mainly remote, voluntary and due to the FLOSS philosophy mostly liberal) but this seems to be one to be celebrated because of the positive conclusion it showed for women.
Given that from this article's stats it even admits there are 15 times more male self-identified programmers then women on github, a 4% difference in pull requests accepted is tiny.
The data is just convenient for anyone with a political agenda to make big claims. If the paper wanted to be genuine they could have gone further - what was rejected? Was there a valid reason? etc
Based on the findings about men, it seems that both men and women experience a drop when they identify gender. Women experience a greater drop than men, but it is worth considering this trend is not found in the treatment of any one gender.
So the question remains, why do both genders experience a drop. Once we answer that, we can look into why women experience a worse drop.
The difference between men and women when their gender is identifiable is 0.5% (63.5 vs 64.0). Lets first have some good discussion on how significant that difference is.
[Edit]
Retracting my comment below, after the parent comment was edited to provide more context.
---
This quote doesn't seem representative of the whole story that is easily discoverable in the abstract.
> Surprisingly, our results show that women's contributions tend to be accepted more often than men's. However, when a woman's gender is identifiable, they are rejected more often. Our results suggest that although women on GitHub may be more competent overall, bias against them exists nonetheless.
This is indeed in their abstract, but it's misleading. Take a look at page 15 of the preprint. Their data shows that gendered profiles have lower acceptance rates for both genders; that is, pull requests from a gender-neutral profile are more likely to be accepted regardless of the gender of the person behind it.
This paper gives the impression that the authors started out hoping to find evidence of discrimination in the form of a lower pull acceptance rate for women than men, they found the opposite, and then they data mined until they could find something they could use to argue there's an anti-woman bias.
Many hypothesis are discussed in an attempt to explain this and the paper is well worth the read. Thanks to a documented GitHub API this stuff gets quite scientifically correct - everything is backed up with real-world data.
More quotes:
> To summarize this paper’s observations:
> 1. Women are more likely to have pull requests accepted than men.
> 2. Women continue to have high acceptance rates as they gain experience.
> 3. Women’s pull requests are less likely to serve an immediate project need.
> 4. Women’s changes are larger.
> 5. Women’s acceptance rates are higher across programming languages.
> 6. Women have lower acceptance rates as outsiders when they are identifiable as women.