Tenured prof here. Every paper of mine goes on Arxiv with no exceptions, published under CC BY-NC-ND licenses. Some of us are working hard to overcome the system (e.g. look at the IACR's efforts). Unfortunately, academics are still hindered by institutional inertia; in fact, many prefer the status quo, usually those who rely on prestige over actual quality to advance their careers.
> usually those who rely on prestige over actual quality to advance their careers
Unfortunately for those of us pre-tenure, it's difficult to balance these as I'm sure you aware. We're evaluated by people who may have the best intentions, but don't work directly in our field. They then determine whether we keep our jobs. It's difficult not to consider prestige as a factor when you know those evaluating you will.
I cannot understand how people think that trying to evaluate people who are pushing the bounds of human knowledge is anything short of "a clusterfuck." I understand the appeal of metrics and prestige, as they provide some signal in an insanely noisy system, but to think they are perfectly aligned is ludicrous. Are we unwilling to just admit that the process is noisy? Is it really not okay that it is (and that we'll probably never escape this aspect)? Because I can't see us doing good science if we're not.
Thank you, thank you, thank you! I've no skin in the game (not an academic and a math idiot but I've a hole in my heart for Aaron Swartz and what he stood for) but I love that there are professors like yourself that believe in the free sharing of knowledge.
What do you mean by the IACR’s efforts here? In the crypto community it’s very much the norm to put everything on eprint, and it is very rare to find a crypto paper not on there
> Unfortunately, academics are still hindered by institutional inertia
As an ABD this has been a real pain point for me. Maybe I came into academia thinking what mattered most was the research. But now I'm the stereotypical PhD who passionately despises academia for its lack of being academic. I'm happy to have competition, but at the end of the day are we all not on the same team?
How the hell did we create a system where it is the norm that an advisor does not read a thesis, to read papers, to mentor? For that to be the norm among a committee? When I've had issues with getting works getting through review (even when they have high citations due to arxiv) I don't understand why it's acceptable for a response to be "keep trying" instead of "here, I read the paper and reviewer responses, let me help"[0]. It seems inefficient that we throw students into the deep end and watch them sink or learn to swim. I think there'd be a lot fewer dejected PhD students if there was a stronger focus on academics, mentorship, and collaboration over churning out ̶w̶i̶d̶g̶e̶t̶s̶ papers.
I think what pisses me off the most is thinking that research significance and success can be measured __purely__ through metrics like citation counts, H-indices, i10's, awards, etc. I'm not saying those are useless, but that we can evaluate without looking at the content? (as you say, actual quality of work) It's like we learned about Goodhart's Law and decided it was a feature not a bug.
(I know this is not always the case and there are many amazing advisors, but I'd be impressed if someone didn't know this is happening at least somewhere within their department.)
[0] If it takes a village to raise a child, it takes a department to mint a PhD. These types of things should come from committees, not just advisors. Our annual meetings and review shouldn't just be going through the motions.
> (...) in fact, many prefer the status quo, usually those who rely on prestige over actual quality to advance their careers.
Your comment doesn't read like one from anyone with any relationship with academia. If you had, you'd know that the issue is not a vacuous "prestige" but funding being dependent on hard metrics such as impact factor, and in some cases with metrics being collected exclusively from a set of established peer-reviewed journals that must be whitelisted.
And ArXiv is not one of them.
This means that a big share of academia has their professional and future, as well as their institution's ability to raise funding, dependent on them publishing on a small set of non-open peer-reviewed journals.
Reading your post, you make it sound like anyone can just upload a random PDF to a random file server and call it a publication. That ain't it. If you fail to understand the problem, you certainly ain't the solution.
> Your comment doesn't read like one from anyone with any relationship with academia.
Your comment reads likewise.
He didn't say he publishes them exclusively on Arxiv. It's quite common for professors to post it there as well as submit to journals. Many (most?) journals allow for it - they don't insist the ones in arxiv be taken down - as long as they're posting preprints and not the final (copyrighted) version.
As an academic, you should also know that practices vary widely with discipline. As an example:
> dependent on them publishing on a small set of non-open peer-reviewed journals.
IIRC, NIH grants require publishing in open peer-reviewed journals.
Also, lots of disciplines are not heavily reliant on funding. In both universities I attended, the bulk of math professors did not even apply for grants! It's not required to get tenure (unlike engineering/physics). Also often true in some economics departments.
As an aside, your comment violates a number of HN guidelines.
FWIW I thought it read like reviewer 2, which makes me actually think they have a relationship with academia.
The problem I have with their comment is that it rejects the critique by pointing to a different issue. As if there is a singular issue with academia that leads to the mess. So it comes off as a typical "reviewer 2" comment to me where the is more complaint and disagreement than critique.
FWIW, I think we in academia need to embrace the fuzziness and noise of evaluation. I think the issue is that by trying to make sense out of an extremely noisy process we placed too much value in the (still noisy) signals we can use. It is a problem to think that these are objective answers and deny the existence of Goodhart's Law (this is especially ironic in ML where any intro class discusses reward hacking). And in this, I think there's a strong coupling between cgshep's and chipdart's complaints.
As for publishing, I think we also lost sight of the main reason we publish: to communicate with our peers. Publishers played an important role since not even a century ago we could not make our works trivially available to our peers. But now the problem is information overload, not lack of information. And I think in part the review process is getting worse and worse each year as we place so little value on the act of reviewing, do not hold anyone accountable for a low quality review[0,1], do not hold ACs or Metas accountable for, and we have so many papers to review that I don't think we can expect high quality reviewing even if we actually incentivized it. I mean in ML we expect a few hundred ACs to oversee over ten thousand submissions?
My question is if we'll learn that the noisiness of the process and the randomness of rejection creates a negative feedback of papers. Where you "roll the dice" on the next conference. You resubmit without changes, as well as your new work (publish or perish baby). If we had quality reviewing at least this would push pressure for papers to get incrementally better instead of just being recycled. But recycling is a very efficient strategy right now and we've seen plenty of data to suggest it is.
[0] I understand the reasons for this. It is a tricky problem
[1] I'd actually argue we incentivize bad reviews. No one questions you when you reject a work and finding reasons to reject a work are far easier to accept one. There's always legitimate reasons to reject any work. Not to mention that the whole process is zero sum, since venue prestige is for some reason based on the percentage of papers rejected. As if there isn't even variance in year to year quality.
But putting your papers on the arXiv, as your parent said, doesn't mean you only put them on the arXiv. I put all my papers on the arXiv, but I also submit them for publication in journals that will help me make the case for funding and promotion.
I all fairness, I don't think the grandparent post disagrees with anything that is in the parent post here.
Yes, academia has tried to quantify prestige via impact factor and peer-reviewed journals. Yes, lots of people (even in Academia) feel that the system is being gamed, with by the publishing houses that own the journals being a common scapegoat.
The system isn't broken, but it also keeps its integrity through some dynamic tension: a bit of criticism is a good thing.
> the issue is not a vacuous "prestige" but funding being dependent on hard metrics such as impact factor
These things are not in contention.
There is no singular problem to be solved, which is why it is so difficult. No smoking gun.
> And ArXiv is not one of them.
ArXiv has a large impact on metrics and so called impact factor. But let's also not be delusional to the fact that a paper from a prestigious institution will always receive more citations (or any other metric) than an equal quality paper from a less prestigious institution. All our metrics can be hacked through publicity.
> Reading your post
Reading yours, it sounds like you stand in the way of resolving issues in academia. Not because you don't have issues with it that you want to solve, but because you have already found the answer.
Your comment reads like someone who has a relationship with academia.
