I think the professional sciences has, for a long time, been a social game of building ones career but it does feel like it's metastasized into something that's swallowed academia.
From the first article in the series [0]:
> Insiders ... understand that a research paper serves ... in increasing importance ... Currency, An advertisement, Brand marketing ... in contrast to what outsiders .. believe, which is ... to share a novel discovery with the world in a detailed report.
I can believe it's absolutely true. And yikes.
Other than the brutal contempt, TFA looks like pretty good advice.
A secondary and less visible consequence of this is that many people don’t go into academia in the first place because they are put off by the publishing system. And so many people that would otherwise be contributing to human knowledge are working in an office somewhere helping a random company sell more widgets.
This is an article about ML research, and the emphasis on branding and marketing your paper wouldn't fly in any of the fields people think of as scientific. Could you imagine someone saying, "be sure that the graphic for the molecule in figure 1 is 3D and has bright colors?"
The most disturbing thing about it is the way advice to forget about science and optimize for the process is mixed with standard tips for good communication. It shows that the community is so far gone that they don't see the difference.
If anyone needs a point of reference, just look at an algorithms and data structures journal to see what life is like with a typical rather than extreme level of problems.
I work in academia and have over 70 papers published.
I Agree with most ideas in the article.
Another dimension not covered is what I called “author engineering”.
Many times it is very difficult to “get into” a new field if you don’t have an author known by the editors. I work in applied math (very transversal) and happen to me often to be rejected because “I don’t belong to the area”.
PhD students usually don’t suffer from this as the supervisor is already a member of the community. But if not, try to bring a collaborator that is known in the area. This is usually done in conferences by chatting with people.
> Many times it is very difficult to “get into” a new field if you don’t have an author known by the editors.
Although there's plenty of critique to go around about the review system, machine learning here typically uses double-blind peer review for the major conferences. That blinding is often imperfect (e.g. if a paper very obviously uses a dataset or cluster proprietary to a major company), but it's not precise enough to reject a paper based on the author being an unknown.
> And it’s not just a pace thing, there’s a threshold of clarity that divides learned nothing from got at least one new idea.
But these days, ideas are quite cheap: in my experience, most researchers have more ideas than students to work on them. Many papers can fit their "core idea" in a tweet or two, and in many cases someone has already tweeted the idea in one form or another. Some ideas are better than others, but there's a lot of "reasonable" ideas out there.
Any of these ideas can be a paper, but what makes it science can't just be the fact that it was communicated clearly. It wouldn't be science unless you perform experiments (that accurately implement the "idea") and faithfully report the results. (Reviewers may add an additional constraint: that the results must look "good".)
So what does science have to do with reviewers' fixation on clarity and presentation? I claim: absolutely nothing. You can pretty much say whatever you want as long as it sounds reasonable and is communicated clearly (and of course the results look good). Even if the over-worked PhD student screws up the evaluation script a bit and the results are in their favor (oops!), the reviewers are not going to notice so long as the ideas are presented clearly.
Clear communication is important, but science cannot just be communicating ideas.
Clarity is and should be absolutely crucial, though.
As an academic I need to be up to date in my discipline, which means skimming hundreds of titles, dozens of abstracts and papers, and thoroughly reading several papers a week, in the context of a job that needs many other things done.
Papers that require 5x the time to read because they're unnecessarily unclear and I need to jump around deciphering what the authors mean are wasting me and many others' time (as are those with misleading titles or abstracts), and probably won't be read unless absolutely needed. They are better caught at the peer review stage. And lack of clarity can also often cause lack of reproducibility when some minor but necessary detail is left ambiguous.
> But these days, ideas are quite cheap: in my experience, most researchers have more ideas than students to work on them.
By “idea” researchers usually imply “idea for a high-impact project that I’m capable of executing”. It’s not just about having ideas, but about having ideas that will actually make an impact on your field. Those again come in two flavors: “obvious ideas” that are the logical next step in a chain of incremental improvements, but that no one yet had time or capability to implement; and “surprising ideas” that can really turn a research field upside down if it works, but is inherently a high-risk/high-reward scenario.
Speaking as a physicist, I find the truly “surprising ideas” to be quite rare but important. I get them from time to time but it can take years between. But the “obvious” ideas, sure, the more students I have the more of them I’d work on.
> Any of these ideas can be a paper, but what makes it science can't just be the fact that it was communicated clearly. It wouldn't be science unless you perform experiments (that accurately implement the "idea") and faithfully report the results. (Reviewers may add an additional constraint: that the results must look "good".)
I kinda agree with this. With the caveat that I’d consider e.g. solving theoretical problems to also count under “experiment” in this specific sentence, since science is arguably not just about gathering data but also developing a coherent understanding of it. Which is why theoretical and numerical physics count as “science”.
On the other hand, I think textbooks and review papers are crucial for science as a social process. We often have to try to consolidate the knowledge gathered from different research directions before we can move forward. That part is about clear communication more than new research.
I think in some ways science has been co-opted by careerists who try to minmax output to accelerate their careers. Being idea obsessed is part of this. It’s much easier to get a paper published that’s on the hype train as opposed to a paper that challenges some idea. Publications justify grant money, grant money justifies more people and more power, more power justifies promotions. And if you talk with early career scientists they all will say they are only doing it until they get a permanent position. Then they will become more curious. Maybe they do, maybe they don’t, I have many older colleagues who are quite curious compared to their younger counterparts. but I believe rewarding ambition at the expense of curiosity is somewhat anti intellectual. It’s sad because I think science should reorganise as the current structure of departments into disciplines may be dated and restructuring could help alleviate this a lot since interdisciplinary work may leverage curiousity over ambition as curiosity will be rewarded with high impact work. But who knows. I can arm chair my way into anything.
The point of a paper isn't "I had this idea" nor is it "I have this evidence". It is "I had this idea, and it turned out to work! (btw here's the evidence I found that convinced me it works).
The value lies in getting true ideas in front of your eyeballs. So communicating the idea clearly is crucial to making the value available.
This person is a clown, probably with a paid agenda, and they should be disallowed from saying such dumb things where smart people with useful skills might read it.
I have a theory that this focus on ideas vs solutions also divides individual researchers, in what drives them. Agreed that academia celebrates and rewards ideas, not solutions. And maybe that’s ok and how it should be, solutions can be done in industry? But the SNR of ideas feels too high at this point.
generating the ideas “planets move at constant per planet velocity” “planets move at a specific speed as a power law function of distance from the sun and we fit the paramets great” “each planet sweeps equal areas in equal time” is cheap, but evaluating which idea is good is expensive, and the whole value of that evaluation is captured in the final idea
This whole take is embarrassingly ignorant and no one with the credentials has the time to check you. We need people to do real thinking and they need to ignore you.
I am not the original author, but I posted this since it mirrors some experiences I have had in my PhD so far submitting papers. This kind of tweaking in paper and writing even happens when writing the first draft or sometimes even in the conception of the research idea or how to go about the implementation and experimentation.
There is a half-joke in our lab that the more times a paper is rejected, the bigger or more praised it will be once it's accepted. This simply alludes to the fact that many times reviewers can be bothered with seeing value in certain ideas or topics in a field unless it is "novel" or the paper is written in a way that is geared towards them, rather than being relegated to "just engineering effort" (this is my biased experience). However, tailoring and submitting certain ideas/papers to venues that value the specific work is the best way I have found to work around this (but even then it takes some time to really understand which conferences value which style of work, even if it appears they value it).
I do think there is some saving grace in the section the author writes about "The Science Thing Was Improved," implying that these changes in the paper make the paper better and easier to read. I do agree very much with this; many times, people have bad figures, poor tables or charts, bad captions, etc., that make things harder to understand or outright misleading. But I only agree with the author to a certain extent. Rather, I think that there should also be changes made on the other side, the side of the reviewer or venue, to provide high-quality reviews and assessments of papers. But I think this is a bit outside the scope of what the author talks about in their post.
There are other posts in the series of the author. He was the co-author of BERT! Yet his paper was scoffed at as "just engineering". He knows what he is talking about.
I have a running joke with my friends. "If your paper is not rejected once, what kind of science are you doing". Either you spent too much time on a project or aiming low.
Thanks for this post. As someone writing an open-source book (without an editor to help), I find some takeaways very helpful.
But I think your most significant change was changing the "what" to "why".
Reading the original, we can see that most sentences start with "we did..." "we did..." and my impression as a reader was, "Okay, but how is this important?" In the second one, the "what" is only in the first part of the sentence, to name things (which gives a sense of novelty), and then only "whys" come after it.
"Whys" > "Whats" also applies to good code comments (and why LLM's code sometimes sucks). I can easily know "what" the code does, but often, I want to know "why" it is there.
I have a problem with this. In the old days, people did research for the sake of research, and mostly out of Europe came the greatest scientific works we have seen. I did my PhD in the US, and it is very unfortunate that "gaming" publications and focusing on "grants" is the meat of research. Before I get criticized, I was part of this process at a top 10 university and I am a proud American. It is because of this pride that I must show tough love. I chose to move away from academia without a postdoc because I hated it. I wanted to do research and contribute to work that pushes my field forward. Most (90% of those I met, and I dare say 99% of international students) only wanted a PhD for selfish reasons (entry to US market, salary bump, changing fields, access to RnD jobs, etc). Perhaps I am naive, but I wish more people did research for the sake of research. The only Clay prize went to a Russian who hated academia. Perhaps there is some truth in the fact the immortals in science are not those churning conference papers, but those laying seeds a la Laplace, Einstein, etc. I want to see more of those, because this is what will move the field forward. It is not manipulating metrics to improve a neural network for one use case, while knowing (and not sharing) it fails in every other instance. This is my second beef with research. When something is tried but does not work, it is not shared. Someone else will try and fail, and this build up will overall slow everyone down. I wish we were more accepting of failed trials, and of not knowing the answer (sharing results without the theory is OKAY. It is OKAY if someone else comes up with it using your results. Having spent many years in a PhD, I can confirm the vast majority unfortunately do not share my point of view. And I hope I do not come across as bitter, it frankly makes me sad.
"In the old days, people did research for the sake of research, and mostly out of Europe came the greatest scientific works we have seen."
In the old days, scientific careers were largely restricted to the independently wealthy or those who could secure patrons.
I also feel like there's a sort of tension with what Hacker News broadly wants out of science. There's often a lament that there aren't enough staff science positions, or positions where people can have a career beyond a postdoc that's just devoted to research.
Those things have to be paid for. Postdocs are expensive. Staff scientists are expensive - and terrifying, because they have careers and kids and mortgages. Postdocs are expensive.
That ends up eating a lot of a PIs time, because the success rate on proposals are low. Even worse now.
Would I love to be able to just sit in my office, think my thoughts, and occasionally write those thoughts up? Sure. But I'd also like to give people an opportunity to have careers in science where they can get paid.
I think a more charitable reading is that these are just basic suggestions about how to make one's writing clear and get your point across. It's hard to step back and look at what you write from the perspective of someone not familiar with the subject matter (ie. the reviewer).
Sure it's framed in terms of "helping you get published" (which feels kind of gross) but I think ultimately it's really about tips for authors to get their points across in a clear and engaging way.
I mean, at some point science is communication. Great for Einstein if he gets general relativity, but if he wants anyone else to care, he needs to communicate not only the complex idea in a clear manner, but also _why_ I should spend my cherished minutes here on earth trying to wrap my small brain around it.
It's the difference between being a Cassandra or the Oracle at Delphi. Maybe the only difference between the two was presentation? (Classicists, feel free to roast my metaphor).
I think it’s equally likely that the second version just got a different set of reviewers who randomly liked it more, and the revisions didn’t make a big difference. Having submitted lots of papers to conferences like this I basically think of the reviewer ratings as noise.
For both grants and papers in my experience, there's a "Doomed"/"Not Doomed" threshold you have to get over, but if you clear that threshold things get fairly stochastic.
For *ACL you'd have to justify your wish to change reviewers, though; and you need a good reason for that. I don't know how much reviewers changes for a resubmission are solely due to reviewers' unavailability but it seems unlikely all three of them got removed from the reviewer pool.
Oh dear... a monkey has escaped from the circus and is telling us the truth about what's going on inside it.
> "The primary objects of modern science are research papers. Research papers are acts of communication. Few people will actually download and use our dataset. Nobody will download and use our model—they can’t, it’s locked inside Google’s proprietary stack."
The author is confusing the concept of 'science as a pursuit that will earn me enough money and prestige to live a nice life' - in which, I'd say, we can replace 'science' with 'religion' and go back to the 1300s or so - with science as the practice of observation, experiment and mathematical theory with the goal of gaining some understanding of the marvelously wonderful universe we exist in.
Yes, the academic system has been grotesquely corrupted by Bayh-Dole, yes, the academic system is internal blood sport politics for a limited number of posts, yes, it's all collapsing under the weight of corporate corruption and a degenerate ruling class - but so what, science doesn't care. It can all go dormant for 100 years, it has before, hasn't it? 125 years ago you had to learn to read German to be up on modern scientific developments.
Wake up - nature doesn't care about the academic system, and science isn't reliant on some decrepit corrupt priesthood.
P.S. Practically speaking, new graduate students should all be required to read Machiavelli as an intro to their new life.
Ot was a fun read, but, how do you know these changes made your paper better? Your assumption is that reviewers approach the reviewing process with the same knowledge and goals, or are quite objective, but that's mot the case in all my publication history. So how can you prove causal effects with 1 sample?
I was surprised to see the author claim that citations in the Introduction are a bad thing. I do think ML papers are generally pretty bad at acknowledging other relevant literature, but this makes me think it’s an active decision somehow
Looking at the differences between the rejected and accepted papers, I don't think it's quite a matter of 'avoiding citations'. The changes seem to break along two lines.
1. Avoid overly general citations. The rejected paper leads with references to image captioning tasks in general and visual question-answering, neither of which is directly advanced by the described study. The accepted paper avoids these general citations in favour of more specific literature that works directly on the image-comparison task.
2. Don't lead with citations. The accepted paper has its citations at the end of the introduction, on page 2.
I think that each change is reasonably justified.
In avoiding overly-general citations, the common practice in machine learning literature is to publish short papers (10 pages or fewer for the main body), and column inches spent in an exhaustive literature review are inches not spent clearly describing the new study.
Placing citations towards the end of the introduction is consistent with the "inverted pyramid" school of writing, most commonly seen in journalism. Leaving the review process out of it for the moment, an ordinary researcher reading the article probably would rather know what the paper is claiming more than what the paper is citing. A page-one that can tell a reader whether they'll be interested in the rest of the article does readers a service.
You should definitely cite, but move the citations to where they are the most relevant and make them specific. The abstract and introduction should be more focused on what you've achieved, and overview of how you've achieved it, and why it is interesting. There generally shouldn't be too much to cite here. The exact details of methods used and everything you've built on comes later in the paper and that is where citations become important and relevant.
My least favourite type of citations in introductions, that I often see from more junior researches are ones that look like:
"In this paper we use a Machine Learning [1][2][3] technique known as Convolutional [4] Neural Networks [5][6][7][8] to..."
I think it would benefit people to look a few layers above themselves, and to see the big picture of the system, who the different actors are, what their goals are, how they are pursuing them etc. Like the "follow the money" game where juniors in corporations are told to try to understand the flow of money and business value and revenue as soon as possible, in order to know how to advance their corporate careers.
In academia the equivalent is prestige. Who gets it and how? Who are the players? There are college students, PhD students, professors, administrators, grant committees, corporation-university industrial collaborations and consortiums, individual managers at corporations and their shareholders, university boards, funding agency managers, politicians allocating taxpayer money to research funding, journal editors, reviewers, tenure committees, pop science magazine editors, pop science magazine readers, general public taxpayers.
You should be able to put yourself in the shoes of each of these and have a rough idea of how they can obtain prestige as input from some other actor and how they can pass on prestige to yet another actor. You must understand the flow of prestige, and then it will be much less mysterious. (Of course understanding the flow of money also helps, but people tend to overlook prestige because one of the least prestigious things is to overtly care about prestige, it's supposed to seem effortless and unacknowledged)
Another strategy is to write a descent paper and submit it somewhere good. If it’s accepted, great. If it’s rejected and the comments make sense, improve the paper based on the comments before resubmitting somewhere else. Otherwise simply resubmit somewhere else.
Most of these are great advises including the other posts in PhD metagame series.
Max should publish this in a book and it will probably sell by truckloads.
If I've to choose by ranking in usefulness, it will probably topic no. 4 is the best part "Don't Make Things Actually Work". Topic no. 3 is the second. This particular topic no. 5 is the third. Topic no.1 is the fourth. The topic no. 2 is the fifth ranking in usefulness but overall great advises nonetheless.
Perhaps the last one for the topic is when and how to wrap up the PhD research since research is a never ending endeavor.
Another Machiavellian thing I have seen in the literature related to "Science 2" is where ML benchmarks or test cases only become accepted when they show that a lot of people's models are working. ;-)
There is saying in english that people "eat with their eyes".
When it comes to papers, I always reminded myself and others that people also _read_ with their eyes.
It is easy to be cynical about this (with some justification!), but if the findings are more clearly and quickly communicated by a pretty-looking paper, then the paper has objectively improved.
Interesting tips, but it also depends on the field.
If you're submitting to a control theory journal, you better have some novel theorems with rigorous mathematical proofs in that "rest of the paper" part. That's a little nontrivial.
Sure, but if you can't articulate why those theorems and proofs are important to be pursued, how it's different from all the related works, what your unique contribution is and why that matters, what the previous works lacked or got wrong, what the impact and value of your work is, i.e. why anyone should care at all, then you'll have a hard time getting it accepted. Just because the math checks out and was difficult and took a lot of effort, it doesn't guarantee that the work is worthy of dissemination.
I was reading this and thinking that the research could be used by LLMs to identify birds using the Birds-to-Words dataset identified in this research paper.
It seems to go 180 degrees against what a smart starry-eyed junior grad student would believe. Surely, it's all about actually making things work, right? We are in the hard sciences, we don't just craft narratives about our ideas, we make cold hard useful things that are objectively and measurably better and can be used by others, building on top of it, standing on our shoulders, and what could be more satisfying than seeing the fruits of our research being applied and used.
However, for an academic career you want to cultivate the profile of a guru, a thought leader, a visionary, a grand ideas person. Fiddling with the details to put a working system together is lowly and kinda dirty work, like fixing clogged toilets or something. Not like the glorious intellectual work of thinking up great noble thoughts about the big picture.
If you want to pivot to industry, it could help you to build a track record of having created working systems, sure. But I've often seen grad students get stuck on developing bepoke internal systems that are not even really visible to potential future employers. Like improving the internal compute cluster tooling, automating the generations of figures in Latex, building a course management system to keep track of assignment submissions and exam grading and so on. Especially when you're at a phase where your research project is getting rejections and you feel stuck, you are most prone to dive into these invisible, career-killing types of work. In academia, what counts is your published research, your networking opportunities obtained through going to conferences where you have papers, getting cold emailed because someone saw your paper etc. I've seen very smart PhD students get stuck in engineering rabbit holes and it's sad. It happens less if your parents were already in academia, and you kinda get the gist of how things work via osmosis. But outsiders don't really grok what actually makes a difference and what is totally invisible (and a waste from a career perspective). Another such trap is pouring insane amounts of hours into teaching assistance and improving the materials, slides, handouts and so on. The careerists will know to spend just as much on this sort of stuff as they absolutely have to. Satisficing, not optimizing. Do enough to meet the bar, and not one minute more. It is absolutely invisible to the wider academic research community whether your tutorial session on Tuesday to those 20 students was stellar or just OK. Winners of the metagame ruthlessly optimize for visible impact and offload everything else to someone else or just not do them. A publication is visible. A research semester at a prestigious university is visible. Getting a grant is visible. Being the organizer of a workshop is visible. Meticulously grading written exams is invisible. Giving a good tutorial session is invisible. Improving the compute infrastructure of the lab is invisible. Being the goto person regarding Linux issues is invisible.
Packaging your research in a way that works well out of the box is in the middle on this spectrum. It may be appreciated by another stressed PhD student somewhere in some other university, and it may save them some time in setting things up. But that other PhD student won't sit on your grant committee or promotion board. So it might as well be invisible. Unless your work is so stellar and above and beyond other things that it goes viral and you become known to the community through it. But it's a double edged sword, because being known for having packaged your work in an easy to use manner will get you pigeonholed into the "software engineer technician" category, and not the "ideas person" category. Execution is useful but not prestigious. Like the loser classmate whose homework gets copied but isn't invited to parties.
The metagame winner recognizes that their work is transient. Any time spent on packaging up the research software for ease of use or ease of reproducibility once the publication is accepted is simply time stolen from the next project that could get you another publication. Since you'll likely improve the performance in the next slice of the salami anyway, there would be no use in releasing that outdated software so nicely. The primary research output is the paper itself, and the talks and posts you can make to market it to boost its citations, as well as the networking opportunities that happen around the poster and the conference. Extras beyond that are nice, but optional.
While you're working on making something "really" work, you're either delaying the publication, making it risky to get scooped (if done before publication), or you're dumping time into a dead project (dead in the sense that the paper is already published and won't be published-er by pouring more time into it post-publication).
I'm a bit afraid that some people will read this article or skim it and say "The fact that you have to do all of this 'branding' is just further proof that science is riddled with irredeemable incentive issues." However, this isn't the author's point. In fact, early the the post, the author writes:
>The tweaks that get the paper accepted—unexpectedly, happily—also improve the actual science contribution.
>The main point is that your paper’s value should be obvious, not that is must be enormous.
This is slightly oversimplified, but from the outside, science may look like researchers are constantly publishing papers sort of for the sake of it. However, the papers are the codified ways in which we attempt to influence the thinking of other researchers. All of us who engage in scientific research aim to be on the literal cutting edge of the research conversation. Therefore it's imperative to communicate how our work can be valuable to specific readers.
Let's take a look at the two abstracts:
(Version 1, Rejected): Given two distinct stimuli, humans can compare and contrast them using natural language. The comparative language that arises is grounded in structural commonalities of the subjects. We study the task of generating comparative language in a visual setting, where two images provide the context for the description. This setting offers a new approach for aiding humans in fine grained recognition, where a model explains the semantics of a visual space by describing the difference between two stimuli. We collect a dataset of paragraphs comparing pairs of bird photographs, proposing a sampling algorithm that leverages both taxonomic and visual metrics of similarity. We present a novel model architecture for generating comparative language given two images as input, and validate its performance both on automatic metrics and visa human comprehension.
Here, the first two sentences a) make a really obvious claim and could equally be at home in a philosophy journal, a linguistic journal, a cognitive science journal, a psychology journal, a neuroscience journal, even something about optometry. Moreover, some readers may look at this abstract and think "well, that's nice, but I'm not sure I need to read this."
(Version 2, Accepted): We introduce the new Birds-to-Words dataset of 41k sentences describing fine-grained differences between photographs of birds. The language collected is highly detailed, while remaining understandable to the everyday observer (e.g., “heart-shaped face,” “squat body”). Paragraph-length descriptions naturally adapt to varying levels of taxonomic and visual distance—drawn from a novel stratified sampling approach—with the appropriate level of detail. We propose a new model called Neural Naturalist that uses a joint image encoding and comparative module to generate comparative language, and evaluate the results with humans who must use the descriptions to distinguish real images. Our results indicate promising potential for neural models to explain differences in visual embedding space using natural language, as well as a concrete path for machine learning to aid citizen scientists in their effort to preserve biodiversity.
Compared to V1, the V2 abstract does a much better job of communicating a) how this project might be valuable to people who want to understand and use neural-network models "to explain differences in visual embedding space using natural language." Or to put it another way, if you want to understand this, it's in your interest to read the paper!
abetusk|9 months ago
From the first article in the series [0]:
> Insiders ... understand that a research paper serves ... in increasing importance ... Currency, An advertisement, Brand marketing ... in contrast to what outsiders .. believe, which is ... to share a novel discovery with the world in a detailed report.
I can believe it's absolutely true. And yikes.
Other than the brutal contempt, TFA looks like pretty good advice.
[0] https://maxwellforbes.com/posts/your-paper-is-an-ad/
keiferski|9 months ago
whatshisface|9 months ago
The most disturbing thing about it is the way advice to forget about science and optimize for the process is mixed with standard tips for good communication. It shows that the community is so far gone that they don't see the difference.
If anyone needs a point of reference, just look at an algorithms and data structures journal to see what life is like with a typical rather than extreme level of problems.
techas|9 months ago
fl4tul4|9 months ago
Majromax|9 months ago
Although there's plenty of critique to go around about the review system, machine learning here typically uses double-blind peer review for the major conferences. That blinding is often imperfect (e.g. if a paper very obviously uses a dataset or cluster proprietary to a major company), but it's not precise enough to reject a paper based on the author being an unknown.
nicce|9 months ago
eeeeeeehio|9 months ago
> And it’s not just a pace thing, there’s a threshold of clarity that divides learned nothing from got at least one new idea.
But these days, ideas are quite cheap: in my experience, most researchers have more ideas than students to work on them. Many papers can fit their "core idea" in a tweet or two, and in many cases someone has already tweeted the idea in one form or another. Some ideas are better than others, but there's a lot of "reasonable" ideas out there.
Any of these ideas can be a paper, but what makes it science can't just be the fact that it was communicated clearly. It wouldn't be science unless you perform experiments (that accurately implement the "idea") and faithfully report the results. (Reviewers may add an additional constraint: that the results must look "good".)
So what does science have to do with reviewers' fixation on clarity and presentation? I claim: absolutely nothing. You can pretty much say whatever you want as long as it sounds reasonable and is communicated clearly (and of course the results look good). Even if the over-worked PhD student screws up the evaluation script a bit and the results are in their favor (oops!), the reviewers are not going to notice so long as the ideas are presented clearly.
Clear communication is important, but science cannot just be communicating ideas.
Al-Khwarizmi|9 months ago
As an academic I need to be up to date in my discipline, which means skimming hundreds of titles, dozens of abstracts and papers, and thoroughly reading several papers a week, in the context of a job that needs many other things done.
Papers that require 5x the time to read because they're unnecessarily unclear and I need to jump around deciphering what the authors mean are wasting me and many others' time (as are those with misleading titles or abstracts), and probably won't be read unless absolutely needed. They are better caught at the peer review stage. And lack of clarity can also often cause lack of reproducibility when some minor but necessary detail is left ambiguous.
setopt|9 months ago
By “idea” researchers usually imply “idea for a high-impact project that I’m capable of executing”. It’s not just about having ideas, but about having ideas that will actually make an impact on your field. Those again come in two flavors: “obvious ideas” that are the logical next step in a chain of incremental improvements, but that no one yet had time or capability to implement; and “surprising ideas” that can really turn a research field upside down if it works, but is inherently a high-risk/high-reward scenario.
Speaking as a physicist, I find the truly “surprising ideas” to be quite rare but important. I get them from time to time but it can take years between. But the “obvious” ideas, sure, the more students I have the more of them I’d work on.
> Any of these ideas can be a paper, but what makes it science can't just be the fact that it was communicated clearly. It wouldn't be science unless you perform experiments (that accurately implement the "idea") and faithfully report the results. (Reviewers may add an additional constraint: that the results must look "good".)
I kinda agree with this. With the caveat that I’d consider e.g. solving theoretical problems to also count under “experiment” in this specific sentence, since science is arguably not just about gathering data but also developing a coherent understanding of it. Which is why theoretical and numerical physics count as “science”.
On the other hand, I think textbooks and review papers are crucial for science as a social process. We often have to try to consolidate the knowledge gathered from different research directions before we can move forward. That part is about clear communication more than new research.
mnky9800n|9 months ago
rocqua|9 months ago
The value lies in getting true ideas in front of your eyeballs. So communicating the idea clearly is crucial to making the value available.
smolder|9 months ago
jhrmnn|9 months ago
QuadmasterXLII|9 months ago
smolder|9 months ago
smolder|9 months ago
stefanpie|9 months ago
There is a half-joke in our lab that the more times a paper is rejected, the bigger or more praised it will be once it's accepted. This simply alludes to the fact that many times reviewers can be bothered with seeing value in certain ideas or topics in a field unless it is "novel" or the paper is written in a way that is geared towards them, rather than being relegated to "just engineering effort" (this is my biased experience). However, tailoring and submitting certain ideas/papers to venues that value the specific work is the best way I have found to work around this (but even then it takes some time to really understand which conferences value which style of work, even if it appears they value it).
I do think there is some saving grace in the section the author writes about "The Science Thing Was Improved," implying that these changes in the paper make the paper better and easier to read. I do agree very much with this; many times, people have bad figures, poor tables or charts, bad captions, etc., that make things harder to understand or outright misleading. But I only agree with the author to a certain extent. Rather, I think that there should also be changes made on the other side, the side of the reviewer or venue, to provide high-quality reviews and assessments of papers. But I think this is a bit outside the scope of what the author talks about in their post.
karel-3d|9 months ago
pks016|9 months ago
jampa|9 months ago
But I think your most significant change was changing the "what" to "why".
Reading the original, we can see that most sentences start with "we did..." "we did..." and my impression as a reader was, "Okay, but how is this important?" In the second one, the "what" is only in the first part of the sentence, to name things (which gives a sense of novelty), and then only "whys" come after it.
"Whys" > "Whats" also applies to good code comments (and why LLM's code sometimes sucks). I can easily know "what" the code does, but often, I want to know "why" it is there.
MPSFounder|9 months ago
Fomite|9 months ago
In the old days, scientific careers were largely restricted to the independently wealthy or those who could secure patrons.
I also feel like there's a sort of tension with what Hacker News broadly wants out of science. There's often a lament that there aren't enough staff science positions, or positions where people can have a career beyond a postdoc that's just devoted to research.
Those things have to be paid for. Postdocs are expensive. Staff scientists are expensive - and terrifying, because they have careers and kids and mortgages. Postdocs are expensive.
That ends up eating a lot of a PIs time, because the success rate on proposals are low. Even worse now.
Would I love to be able to just sit in my office, think my thoughts, and occasionally write those thoughts up? Sure. But I'd also like to give people an opportunity to have careers in science where they can get paid.
geokon|9 months ago
Sure it's framed in terms of "helping you get published" (which feels kind of gross) but I think ultimately it's really about tips for authors to get their points across in a clear and engaging way.
jszymborski|9 months ago
It's the difference between being a Cassandra or the Oracle at Delphi. Maybe the only difference between the two was presentation? (Classicists, feel free to roast my metaphor).
canjobear|9 months ago
Fomite|9 months ago
spidersouris|9 months ago
photochemsyn|9 months ago
> "The primary objects of modern science are research papers. Research papers are acts of communication. Few people will actually download and use our dataset. Nobody will download and use our model—they can’t, it’s locked inside Google’s proprietary stack."
The author is confusing the concept of 'science as a pursuit that will earn me enough money and prestige to live a nice life' - in which, I'd say, we can replace 'science' with 'religion' and go back to the 1300s or so - with science as the practice of observation, experiment and mathematical theory with the goal of gaining some understanding of the marvelously wonderful universe we exist in.
Yes, the academic system has been grotesquely corrupted by Bayh-Dole, yes, the academic system is internal blood sport politics for a limited number of posts, yes, it's all collapsing under the weight of corporate corruption and a degenerate ruling class - but so what, science doesn't care. It can all go dormant for 100 years, it has before, hasn't it? 125 years ago you had to learn to read German to be up on modern scientific developments.
Wake up - nature doesn't care about the academic system, and science isn't reliant on some decrepit corrupt priesthood.
P.S. Practically speaking, new graduate students should all be required to read Machiavelli as an intro to their new life.
3abiton|9 months ago
mobeets|9 months ago
Majromax|9 months ago
1. Avoid overly general citations. The rejected paper leads with references to image captioning tasks in general and visual question-answering, neither of which is directly advanced by the described study. The accepted paper avoids these general citations in favour of more specific literature that works directly on the image-comparison task.
2. Don't lead with citations. The accepted paper has its citations at the end of the introduction, on page 2.
I think that each change is reasonably justified.
In avoiding overly-general citations, the common practice in machine learning literature is to publish short papers (10 pages or fewer for the main body), and column inches spent in an exhaustive literature review are inches not spent clearly describing the new study.
Placing citations towards the end of the introduction is consistent with the "inverted pyramid" school of writing, most commonly seen in journalism. Leaving the review process out of it for the moment, an ordinary researcher reading the article probably would rather know what the paper is claiming more than what the paper is citing. A page-one that can tell a reader whether they'll be interested in the rest of the article does readers a service.
dagw|9 months ago
My least favourite type of citations in introductions, that I often see from more junior researches are ones that look like:
"In this paper we use a Machine Learning [1][2][3] technique known as Convolutional [4] Neural Networks [5][6][7][8] to..."
bonoboTP|9 months ago
In academia the equivalent is prestige. Who gets it and how? Who are the players? There are college students, PhD students, professors, administrators, grant committees, corporation-university industrial collaborations and consortiums, individual managers at corporations and their shareholders, university boards, funding agency managers, politicians allocating taxpayer money to research funding, journal editors, reviewers, tenure committees, pop science magazine editors, pop science magazine readers, general public taxpayers.
You should be able to put yourself in the shoes of each of these and have a rough idea of how they can obtain prestige as input from some other actor and how they can pass on prestige to yet another actor. You must understand the flow of prestige, and then it will be much less mysterious. (Of course understanding the flow of money also helps, but people tend to overlook prestige because one of the least prestigious things is to overtly care about prestige, it's supposed to seem effortless and unacknowledged)
unknown|9 months ago
[deleted]
speedgoose|9 months ago
teleforce|9 months ago
Max should publish this in a book and it will probably sell by truckloads.
If I've to choose by ranking in usefulness, it will probably topic no. 4 is the best part "Don't Make Things Actually Work". Topic no. 3 is the second. This particular topic no. 5 is the third. Topic no.1 is the fourth. The topic no. 2 is the fifth ranking in usefulness but overall great advises nonetheless.
Perhaps the last one for the topic is when and how to wrap up the PhD research since research is a never ending endeavor.
whatshisface|9 months ago
disqard|9 months ago
"Is the scientific paper a fraud?"
I found a PDF online here: https://www.weizmann.ac.il/mcb/alon/sites/mcb.alon/files/use...
JR1427|9 months ago
When it comes to papers, I always reminded myself and others that people also _read_ with their eyes.
It is easy to be cynical about this (with some justification!), but if the findings are more clearly and quickly communicated by a pretty-looking paper, then the paper has objectively improved.
ubj|9 months ago
If you're submitting to a control theory journal, you better have some novel theorems with rigorous mathematical proofs in that "rest of the paper" part. That's a little nontrivial.
bonoboTP|9 months ago
firesteelrain|9 months ago
fl4tul4|9 months ago
unknown|9 months ago
[deleted]
bonoboTP|9 months ago
It seems to go 180 degrees against what a smart starry-eyed junior grad student would believe. Surely, it's all about actually making things work, right? We are in the hard sciences, we don't just craft narratives about our ideas, we make cold hard useful things that are objectively and measurably better and can be used by others, building on top of it, standing on our shoulders, and what could be more satisfying than seeing the fruits of our research being applied and used.
However, for an academic career you want to cultivate the profile of a guru, a thought leader, a visionary, a grand ideas person. Fiddling with the details to put a working system together is lowly and kinda dirty work, like fixing clogged toilets or something. Not like the glorious intellectual work of thinking up great noble thoughts about the big picture.
If you want to pivot to industry, it could help you to build a track record of having created working systems, sure. But I've often seen grad students get stuck on developing bepoke internal systems that are not even really visible to potential future employers. Like improving the internal compute cluster tooling, automating the generations of figures in Latex, building a course management system to keep track of assignment submissions and exam grading and so on. Especially when you're at a phase where your research project is getting rejections and you feel stuck, you are most prone to dive into these invisible, career-killing types of work. In academia, what counts is your published research, your networking opportunities obtained through going to conferences where you have papers, getting cold emailed because someone saw your paper etc. I've seen very smart PhD students get stuck in engineering rabbit holes and it's sad. It happens less if your parents were already in academia, and you kinda get the gist of how things work via osmosis. But outsiders don't really grok what actually makes a difference and what is totally invisible (and a waste from a career perspective). Another such trap is pouring insane amounts of hours into teaching assistance and improving the materials, slides, handouts and so on. The careerists will know to spend just as much on this sort of stuff as they absolutely have to. Satisficing, not optimizing. Do enough to meet the bar, and not one minute more. It is absolutely invisible to the wider academic research community whether your tutorial session on Tuesday to those 20 students was stellar or just OK. Winners of the metagame ruthlessly optimize for visible impact and offload everything else to someone else or just not do them. A publication is visible. A research semester at a prestigious university is visible. Getting a grant is visible. Being the organizer of a workshop is visible. Meticulously grading written exams is invisible. Giving a good tutorial session is invisible. Improving the compute infrastructure of the lab is invisible. Being the goto person regarding Linux issues is invisible.
Packaging your research in a way that works well out of the box is in the middle on this spectrum. It may be appreciated by another stressed PhD student somewhere in some other university, and it may save them some time in setting things up. But that other PhD student won't sit on your grant committee or promotion board. So it might as well be invisible. Unless your work is so stellar and above and beyond other things that it goes viral and you become known to the community through it. But it's a double edged sword, because being known for having packaged your work in an easy to use manner will get you pigeonholed into the "software engineer technician" category, and not the "ideas person" category. Execution is useful but not prestigious. Like the loser classmate whose homework gets copied but isn't invited to parties.
The metagame winner recognizes that their work is transient. Any time spent on packaging up the research software for ease of use or ease of reproducibility once the publication is accepted is simply time stolen from the next project that could get you another publication. Since you'll likely improve the performance in the next slice of the salami anyway, there would be no use in releasing that outdated software so nicely. The primary research output is the paper itself, and the talks and posts you can make to market it to boost its citations, as well as the networking opportunities that happen around the poster and the conference. Extras beyond that are nice, but optional.
While you're working on making something "really" work, you're either delaying the publication, making it risky to get scooped (if done before publication), or you're dumping time into a dead project (dead in the sense that the paper is already published and won't be published-er by pouring more time into it post-publication).
ke88y|9 months ago
This won’t get you a Stanford professorship. That’s something you can cry about from your mountain chalet or beachfront vacation home.
amelius|9 months ago
unknown|9 months ago
[deleted]
cadamsdotcom|9 months ago
calrain|9 months ago
YossarianFrPrez|9 months ago
>The tweaks that get the paper accepted—unexpectedly, happily—also improve the actual science contribution. >The main point is that your paper’s value should be obvious, not that is must be enormous.
This is slightly oversimplified, but from the outside, science may look like researchers are constantly publishing papers sort of for the sake of it. However, the papers are the codified ways in which we attempt to influence the thinking of other researchers. All of us who engage in scientific research aim to be on the literal cutting edge of the research conversation. Therefore it's imperative to communicate how our work can be valuable to specific readers.
Let's take a look at the two abstracts:
Here, the first two sentences a) make a really obvious claim and could equally be at home in a philosophy journal, a linguistic journal, a cognitive science journal, a psychology journal, a neuroscience journal, even something about optometry. Moreover, some readers may look at this abstract and think "well, that's nice, but I'm not sure I need to read this." Compared to V1, the V2 abstract does a much better job of communicating a) how this project might be valuable to people who want to understand and use neural-network models "to explain differences in visual embedding space using natural language." Or to put it another way, if you want to understand this, it's in your interest to read the paper!rafelolszewski|9 months ago
[deleted]