Merge remote-tracking branch 'origin/leetcode'

Promote draft to post

Author: john-kurkowski

src/content/posts/2025-04-03-still-measuring-devs-with-leetcode.mdx

@@ -0,0 +1,183 @@
+---
+layout: ../../components/layouts/post.astro
+title: 'Still Measuring Devs With LeetCode?'
+categories: [tech]
+tags: [engineering, programming]
+description:
+  I'm disappointed to find tech interviews in 2025 are still asking LeetCode
+  questions. Companies haven't listened to old advice on competitive hiring like
+  DHH's "Hiring programmers with a take-home test" or Matasano Security's "the
+  Hiring Post".
+---
+
+I'm disappointed to find tech interviews in 2025 are still asking LeetCode
+questions. Companies haven't listened to old advice on competitive hiring like
+DHH's [Hiring programmers with a take-home test] or Matasano Security's [the
+Hiring Post][The Hiring Post].
+
+> Years from now, we'll look back at the 2015 developer interview as an
+> anachronism, akin to hiring an orchestra cellist with a personality test and a
+> quiz about music theory rather than a blind audition.
+
+Even new companies with the chance of a fresh, competitive approach are still
+asking LeetCode questions, probably because the FAANG-equivalent companies of
+decades ago were asking LeetCode questions. A few FAANGs still are. A
+100-employee startup primed me with an outline of their upcoming interview steps
+with a link to a sample [leetcode.com][LeetCode] question to expect, boasting,
+"Our interview process is similar to Meta's process."
+
+> The most dangerous phrase a \[manager\] can use is "We've always done it that
+> way."
+>
+> —Grace Hopper
+
+In the most recent 10 years of my tech career, I didn't have to interview for
+jobs: behold, the power of referral! Not interviewing meant no LeetCode
+questions. The last time I had to answer a LeetCode question, the name LeetCode
+didn't exist, although the interview style was decades old.
+
+I actually used to be pretty good at LeetCode questions. In my college computer
+science degree, senior year, I went through 20 interview rounds with 12
+different companies. Only the final company accepted me. I didn't get better at
+programming during that time, though that would have been a good use of my time.
+Rather, in that time, I just got better at interviewing. A lot of that was
+practicing LeetCode-style questions, via [the first edition of Programming
+Interviews Exposed][Programming Interviews Exposed]. Having little professional
+experience, I accepted there was no other practice for me to gain companies'
+trust.
+
+Within a few years in the industry, when it was my turn to play interviewer, I
+became empathetic to candidates. LeetCode hazing was an inaccurate and unfair
+way to measure. What was the ritual predicting? What hostility was it
+engendering? I wanted to shake free of the cult. What if we measured actual
+developer skill: can the candidate do the job well, rather than can the
+candidate do LeetCode well? What if a candidate was a great fit for the job, but
+the specialized simulacrum and pressure cooker of live-coding LeetCode
+interviews excluded such a candidate? I sought the advice of the posts at the
+beginning of this article, to end the cycle.
+
+> If you suffered in life and want other people to suffer as you did because
+> "you turned out fine," you did not in fact turn out fine.
+>
+> —@daniel_swensen
+
+Improving the situation was and is easier said than done. Hiring is hard. My
+teams iterated for years to 1) structure our interviews—a structured interview
+is fairer but sadly not the friendliest candidate experience, for the
+interviewer to read from a script _at_ the candidate—and 2) develop relevant,
+realistic, self-contained assessment projects. For take-home assessments, we
+didn't pay candidates for their time, which was tantamount to free labor and
+exclusionary in its own regard. We also lost a great candidate who was a
+prolific OSS contributor to our company's tech stack. They diplomatically
+pointed to their public contributions as evidence of their skill and
+collaboration style, and declined the take-home as something they couldn't
+afford. What were all their free labor OSS contributions for, after all? Our
+team didn't have the capacity to develop and maintain a "fair" backup
+measurement for the special case. While our fairer structures and take-home
+assessment were promising for hiring, we still could've done better.[^1]
+
+During the last few years, I wound down my consulting workload and took a career
+break from tech. Then I wanted to go full-time again in my old full stack web
+developer role, in a new domain. When I started the tech job hunt 12 months ago,
+I no longer had the power of a referral. So I braced for interviews.
+
+I program every day. I know I can do the job. If an interviewer talks to me, I
+can demonstrate I know how to program, I'm emotionally intelligent, and that
+their team would want to work with someone friendly like me. Right?
+
+Much to my chagrin, I didn't make it far in the interview process, because
+almost every company I've interviewed with this year still uses LeetCode-style
+interviews. If a company even gives the heads up of what to expect in an
+interview, don't necessary believe, "Our live code assessment uses real-world
+problems." In 9/10 of my interviews this year, the technical step was still
+LeetCode in a trenchcoat.
+
+I was rusty at these live coding interviews, because in my 14 year career,
+LeetCode and what it teaches _never came up_ on the job. I didn't reach for
+binary trees, stacks, queues, or linked lists. Software development is not
+typically done in a eye-movement surveilled, time-constrained screen-share with
+1-2 others silently breathing down your neck, letting you squirm. Maybe the
+interviewers offer a modicum of hints that make sense to them, as they've seen
+the exact problem several times. Then there's no time left for me to ask them
+questions about the job. The interview for mutual fit turns out not to be a
+2-way street. It's dehumanizing for the experienced dev.
+
+When I'm rejected after an interview, in my spirit of continual improvement, I
+ask for feedback why. 9/10 companies are cagey, to cover their ass. So I'm not
+fully certain all the reasons I was rejected over the last 12 months. I might
+have a worse personality than I think. Maybe a member of the hiring panel was
+underslept and cranky. All I know is, the interview process ending _correlated_
+after I failed to finish a LeetCode step of the interview, i.e. not finishing
+the coding puzzle prompt within 45 minutes.
+
+Roy Lee, developer of [a tool to cheat on these sort of
+interviews][Hard Fork: AI Action Plans], summarized what I was _supposed_ to do.
+
+> You're given 45 minutes and the task here is to _have_ seen the problem before
+> … and be able to regurgitate the memorized solution, without acting like you
+> have seen the problem before.
+
+Some candidates spend months grinding leetcode.com to reach the top 1%, so they
+may be lucky enough to have already seen the 1 interview puzzle close to the
+1,000s they could be given. Other candidates use cheating software like Lee's.
+Meanwhile, experienced developers who could do the actual job well but haven't
+memorized the puzzles struggle to complete them under pressure.
+
+The unqualified candidate aces the test. The qualified candidate fails to finish
+in time, though they can explain what they _would_ do next, given more time. For
+your overloaded hiring pipeline, which candidate would you advance? The test ace
+or the struggling explainer?
+
+Lee answered whether he had ethical concerns about his product.
+
+> LeetCode interviews are about as correlated \[to the job] as, "How many
+> jumping jacks can you do?" … If you were just told to do as many jumping jacks
+> as you could and the winner gets a position, I wouldn't really care if someone
+> was cheating their way through jumping jacks.
+
+Ethics aside, AI-assisted coding tools are great at solving LeetCode problems.
+They have memorized 1,000s of such puzzles. If something is easily solvable by
+computers, maybe that's another hint it's not a great test of a software
+developer? While I haven't subscribed to Lee's product, I know I like using
+AI-assisted coding in my normal job. It would be realistic to have normal tools
+available in an assessment.
+
+While the debate of allowed tools in an assessment rages on, to skip the debate
+next time I'm on the job hunt, I hope to cross-reference my applications with
+the [Hiring Without Whiteboards] list. Companies on the list don't make
+qualified devs jump through hoops with puzzles like LeetCode. The only reason I
+didn't use the list this time? The 2025 tech hiring market is very
+difficult.[^2] I felt beggars couldn't be choosers.
+
+I hope more companies think harder who they're looking for. They're missing out
+on qualified devs by following what's trendy. They're dehumanzing candidates.
+There are better ways to interview and measure. I hope to see more companies one
+day make the Hiring Without Whiteboards list.
+
+### Footnotes
+
+[^1]:
+    Read more about the ethics and meritocracy myth in [Why GitHub is not your
+    CV].
+
+[^2]:
+    The confluence of difficulties in the tech hiring market in 2015 vs. 2025:
+    the magic-promising whisper of "AI" in leadership's ears, venture capital
+    interest rates, and overhiring from the early COVID-19 pandemic, resulting
+    in big tech layoffs and an overflowing candidate pool. In 12 months, I
+    applied to 200+ companies. Less than 10% gave a human response. 50% sent an
+    automated rejection from a "no-reply" email address, within 1-3 months. The
+    rest never tell you the result of your application either way.
+
+[Hard Fork: AI Action Plans]:
+  https://www.nytimes.com/2025/03/21/podcasts/hardfork-ai-action-plans.html
+[Hiring Without Whiteboards]:
+  https://github.com/poteto/hiring-without-whiteboards
+[Hiring programmers with a take-home test]:
+  https://signalvnoise.com/svn3/hiring-programmers-with-a-take-home-test/
+[LeetCode]: https://leetcode.com
+[Programming Interviews Exposed]:
+  https://archive.org/details/programminginter0000mong
+[The Hiring Post]: https://sockpuppet.org/blog/2015/03/06/the-hiring-post/
+[Why GitHub is not your CV]:
+  https://blog.jcoglan.com/2013/11/15/why-github-is-not-your-cv/