hckrnws

I wish the original title was kept ("How to win a best paper award"). It seems a good list for that.

Most of this list is about how to dress for senpai; figuratively speaking. A pretty depressing take on "how to do important research that matters".

I would hope that would be of the most unimportant part of science, totally irrelevant to what's important and what matters. But maybe that's not true today.

> Most of this list is about how to dress for senpai; figuratively speaking. A pretty depressing take on "how to do important research that matters".

How is that the case? The tips seem to aim for impactful research: picking good ideas and executing well on them. There's a tacit assumption that such impactful research will win best paper awards, but that's actually not substantiated and isn't obviously correct, since best paper selection committees can't see the future. For example, many (maybe most?) winners of retrospective awards (test-of-time / influential paper) aren't papers that won a best paper award when originally published.

Most of the author's papers he cites in the post, including the membership inference paper which is one of the papers the author is "most proud of," didn't win best paper awards.

Step 1. Write a paper with Shamir

Step 2. ???

Step 3. Receive award

For non-CS people – if you're a little confused by "conference paper", CS is a little idiosyncratic in that papers are often primarily disseminated through conferences, rather than independent journals. The advice is good in general, though!

Yeah, and I always find the phrase "publish in a conference" to sound vaguely oxymoron-ish.

Well, it's published in the conference's proceedings, and presented at the conference.

https://en.wikipedia.org/wiki/Proceedings

Best bit:

Put in an unreasonable amount of effort

> Earlier I made an analogy to being an explorer; here's another I like even more. Think of yourself as a wildlife photographer. Obviously you need to be in the right place (you won't get a great picture of anything from your couch) and you need to be skilled at your craft. But once you've met those preconditions, the way to get the best picture is to just spend an unreasonable amount of time waiting for exactly the right circumstances to arise.

> The only rule that matters is delivering the message to your reader

The point is he is a superb writer in my book, it is fun to read and there are similarly good quotes that say the same in different ways. It is not enough to hone your technical skill alone because "the right circumstances" do not appear if you only change one variable.

When I read this I see someone having fun, being able to convey that is a good trait.

I’ve always thought the issue was a bit less “Find the interesting research problem” and more “Find the resources, network, or skills that get you into the position of being able to work on the interesting research problem.”

If you asked a bunch of researchers working on the “boring” stuff to predict what the hot papers of the year will be about, do we really think they’ll be that far off base? I’m not talking about groundbreaking or truly novel ideas that seem to come out of nowhere, but rather the high impact research that’s more typical of a field.

Even in big tech companies, it’s quite obvious what the interesting stuff to work on is. But there are limited spots and many more people who want those spots than are available.

Interesting. I don't quite agree. It's one thing to predict what general topics will be hot and popular this year. But that's not the same as what particular research problem will be important and have lasting influence.

There are a few kinds of important research. One is solving a well-defined, well-known problem everyone wants to solve but nobody knows how. Another is proposing a new problem, or a new formulation of it, that people didn't realize was important.

There is also highly-cited research that isn't necessarily important, such as being the next paper to slightly lower a benchmark through some tweaks (you get cited by all the subsequent papers that slightly lower the benchmark even further).

In the book The Cuckoo's egg, Cliff Stoll talks to I think Luiz Alvares. I don't have the book handy here, but Alvarez basically told him to nit get distracted by grants, bosses, ... Here is interesting science to do, so go for it. Just run faster than the rest of the world.

In a way, it was a sidetrack of the book, but for me the attitude speaking from that text was interesting and inspiring. When I could pull it off, it tended to work.

You made me order The Cuckoo's Egg. Luis Alvarez is my scientific hero since I read his memoir last year. Truly underappreciated in the pop-sci community.

I often find this kind of advice too vague to really be useful. “Have taste” in the problems you work on isn’t very actionable. (Unless perhaps you list examples of good and bad taste.)

I’ll admit that I may just be immature at research as almost all my experience has either been attempting to replicate research or to put it into practice in production systems.

"Having taste" is mostly about predicting the future. Which problems are worth studying, which problems you are capable of solving, and which solutions turn out to be important, in retrospect. If there was an actionable way of developing taste in something, the activity itself would probably be so predictable that it would not be a particularly good research topic.

Taste is mostly about having a good intuition on the topics where your intuition is worth following. It tends to develop with experience. But if you want to develop the kind of taste that helps picking good research topics, you need the right kind of experience for that field of research. Experience that turns out to be of the right kind, in retrospect. If your experiences and interests align (again in retrospect), you will probably develop a good taste for research problems in your field of interest. But that requires some amount of luck, in addition to everything else.

That seems even less actionable, and somewhat misaligned with the OP article. “Taste” implies an ability to distinguish between a good example and a bad one. If it’s only recognizable in retrospect then it’s just another name for survivorship bias.

If you need actionable guidelines, you may not be the right person to do research. At least not now.

Research is all about studying topics of uncertain value. You have to commit to a project long before you can say if it's actually worth doing.

Taste comes with deliberate effort and experience. It doesn't tell you that a topic is definitely worth studying, but it increases the likelihood that you will guess right.

What is the point of writing the prescription to “have good taste” then?

Either the reader already has it, in which case there’s no point in being told that. Or the reader doesn’t, in which case you have declared that good taste cannot be taught.

Perhaps the author’s next article should be How to win the lottery: be lucky which is just about as actionable.

The author's answer was:

> But if I had to summarize it in one sentence, it would be that taste comes from practicing the skill of research, keeping your focus always on identifying what works and what doesn't.

Instead of following general guidelines, focus on figuring out what works and what doesn't in each specific situation. Keep doing that for many years, and your taste will develop. Remember that you are training your intuition, not developing a set of exact rules.

It's helpful to tell people that they are in uncharted territory and can't rely on running on autopilot even if you don't have a new map to give them. Whether they can make their way or not is unclear, but the first step is just making sure they understand that they're now in a place where they need to make their own way and can't fully rely on existing maps. Otherwise they might not even realize they need to start asking "am I doing the right thing right now" by themselves.

Taste isn't retroactive pattern matching though. In practice it's closer to "how fast can you tell if something is a dead end." The researchers and builders I've seen with actual taste don't pick winners, they kill losers faster. They have a feel for which problems have tractable surface area and which ones are going to eat 3 years of your life with nothing to show for it.

Taste is not using LLMs to write HN comments.

Unless you're too early in which case you get mocked for decades.

Unrelated, but I see the use of the phrase "taste" as having a strong Twitter / e/acc smell (in a negative way).

I tend to associate it with folks who are prepared to victim blame researchers for not adapting to the "new economy" as being people who have "bad taste" or "low agency", maybe as a way to rationalize/justify the upcoming inqeuality that AI will create.

Basically a recycling of the way "IQ"/smarts/hard-work has historically been used to justify disproportionate rewards for the upper class.

(Obviously a gigantic stretch on my part, and not saying the author is in this camp, but just wanted to vent somewhere)

Taste has a much longer pg history: "Taste for Makers" in 2002, "How Art Can Be Good" in 2006, and "Is There Such a Thing as Good Taste" in 2021.

(If we're venting about words, I'll bring up "opinionated", which has somehow become a positive .)

Links: https://paulgraham.com/taste.html https://www.paulgraham.com/goodart.html https://paulgraham.com/goodtaste.html

Yes this does not work as well for math or physics. The biggest problem in math is arguably the Riemann Hypothesis . Good luck getting up to the speed on the literature on that . You can invest a lifetime trying to solve the biggest problems in physics or math and get nowhere. You may have to choose more modest goals.

Very insightful! I found the section on killing papers to be a helpful reminder. As a Ph.D. student, this can be particularly challenging as your environment expects somewhat steady progress (annual reviews, advisor meetings, etc.), and you're encouraged to finish papers rather than starting over.

This might also be of interest: https://karpathy.github.io/2016/09/07/phd/

The article was so stunningly sanewashed & level-headed, I struggled to identify what in it may cause disagreement, or otherwise justify the label of 'opinionated'.

But the comments have proven me wrong.

[dead]

The actual title is "How to win a best paper award", which is quite different from doing "important research that matters". Most researchers work in very niche and specialized fields, sometimes for their whole life. They grant themselves all sorts of awards within their community, but it doesn't mean their research "matters".

The subtitle is “or, an opinionated take on how to do important research that matters”.

> Another reason ignoring the literature can be helpful is that sometimes a bunch of work tries to solve some problem, and so everyone assumes it must be hard---just because no one has solved it yet, even though no one has really tried a fundamentally different approach

How does one approach collaborators in this situation? Like, hey, I have this idea that solves the problem you have been trying to solve in a fundamentally different way that invalidates all the legacy approaches you have invested in, BTW. My emails that follow this spirit tend to get ghosted.

Sometimes you don't need a collaborator if you have the idea. If the other party is not at all working on the angle that you're interested in, it's probably not the correct collaboration to ask to.

Also, a collaborator is usually not a stranger over the internet, it's often someone who you know and you already worked with, so it is not that ackward to expose a new idea and propose to work together.

It takes time and social skills to make long lasting collaborations, the two parties must trust each other in order to collaborate. In this context, exchanging ideas is not really an issue.

I felt this was why math textbooks pose open problems without mentioning them as such. I feel Karatsuba for example was able to come up with his multiplication scheme because he wasn't aware a mathematician had written a false result proving impossibility of faster than quadratic time multiplication.

Or more concretely that famous story where a student had solved certain open problems in statistics thinking they were homework problems.

This is an exceptional read.

[dead]

[dead]