|
| 1 | +--- |
| 2 | +title: How to learn hard concepts and algorithms |
| 3 | +author: Andrew Moon |
| 4 | +pubDatetime: 2025-10-09T21:00:00+09:00 |
| 5 | +postSlug: 2025-10-09-how-to-learn-hard-concepts-and-algorithms |
| 6 | +featured: true |
| 7 | +draft: false |
| 8 | +tags: |
| 9 | + - Software Engineering |
| 10 | + - Learning |
| 11 | +description: Unveiling the black boxes of complex algorithms |
| 12 | +--- |
| 13 | + |
| 14 | +## Table of Contents |
| 15 | + |
| 16 | +--- |
| 17 | + |
| 18 | +# The Art of Black Boxes |
| 19 | + |
| 20 | +We live in an era where information so complex that it seems impossible to understand by a single reading pass is pretty much everywhere. If you have any sort of experience trying to reach out of your comfort zone and learn something above beginner level, you will have encountered this situation: |
| 21 | + |
| 22 | +> "This seems really hard to understand what's going on, but I guess I can just use this library / framework / algorithm / concept without fully understanding it. I can just treat it as a black box for now." |
| 23 | +
|
| 24 | +And most of the times this is a perfectly fine approach for a vast majority of things. With how modern technology is, the rabbit hole of information essentially never ends, and I'm certain that the average reader here wouldn't want to learn the inner workings of how a transistor works just to be able to do a simple google search. |
| 25 | + |
| 26 | +However, there are these very rare occasions where you really want to deeply understand a certain concept to the point where you can confidently explain it to others of similar or superior skill level. From my experience, this happens in several scenarios: |
| 27 | + |
| 28 | +- Solving a complex optimization problem(All engineering problems / research are optimization problems to a certain degree) |
| 29 | +- Preparing for a technical job interview |
| 30 | +- Just a spark of curiosity |
| 31 | + |
| 32 | +Suddenly, the somewhat lazy black box approach that you've been using everywhere doesn't cut it anymore. You want to understand the inner workings of this complex concept, **_you need to understand it now_**, but you don't know where to start. The documentation is either too high level or pretty much non-existent, the research papers are written in the most obtuse way possible, and the code is an absolute pythonic mess with no type hints or comments. What do you do? |
| 33 | + |
| 34 | +Well, unless you're lucky and one of the above three materials seems actually helpful, you're pretty much screwed. Better admit that you're too dumb to understand it at this point and move on to something else, right? |
| 35 | + |
| 36 | +Sadly, you are indeed screwed. However, there are things you can do to help yourself so that you don't arrive in the same situation again in the future, without having to [understand group theory just to use a segment tree](https://en.wikipedia.org/wiki/Semigroup). |
| 37 | + |
| 38 | +## Does that mean black boxes are bad? |
| 39 | + |
| 40 | +Absolutely not. It would take eons to learn something new if you had to understand everything from the ground up. It is natural that you treat certain complex parts of your learning as black boxes, and it is a very important skill to be able to identify which parts of your learning you can treat as a black box and which parts you need to deeply understand. In fact, as you go further into learning a certain concept, usually you will be able to differentiate between the two more easily as you encounter more references. Let's look at some examples:(feel free to skip them if you know what I'm trying to say) |
| 41 | + |
| 42 | +### Machine Learning - The GPU |
| 43 | + |
| 44 | +Many machine learning researchers and practitioners are often very knowledged in various machine learning algorithms and techniques in their field, which are usually implemented in a way that becomes primitive instructions that are executed on GPUs. I am absolutely **dead certain** that you would have met someone in your life who calls themselves somewhat of an expert in a subfield of machine learning that spits all sorts of unknown jargon about the latest research, but when you actually see how they're actually implementing their ideas into code, they themselves often don't fully understand what's going on as well. To be clear, there are also numerous experts in the field who truly do know what they're doing, but with the hype of machine learning, there are just as many, if not more, people who just ride the wave without fully understanding it. |
| 45 | + |
| 46 | +These sort of 'black boxes' that many machine learning programmers do not fully understand usually originate from either the math behind the algorithm, or the hardware that the algorithm is executed on: the GPU. The math part is often stressed as important, so many such practitioners are conscious of themselves as to whether they understand it or not. The GPU side of things however often gets overlooked. Of all these people talking all sorts of jargon about machine learning, how many would you expect to be able to actually translate their high level python code into CUDA kernels that run at maximum efficiency, aware of the underlying GPU hardware architecture? |
| 47 | + |
| 48 | +Well according to NVIDIA themselves at [GTC 2025](https://www.youtube.com/watch?v=GmNkYayuaA4), not many(<10%). In fact, if you were to look through several SoTA kernel libraries such as [FlashAttention](https://github.com/Dao-AILab/flash-attention), you may easily find several comments from the authors themselves saying things such as how [backwards kernels are a massive pain to implement efficiently](https://github.com/Dao-AILab/flash-attention/issues/1172), or that [there are less than 100 people in the world who can write such performant kernels](https://x.com/Stone_Tao/status/1968168902112002322). Of course, understanding how the GPU works and writing SoTA performance kernels require vastly different levels of skill, but you get the point: most people treat the underlying hardware as a black box, and just use high level libraries such as PyTorch or TensorFlow to do their work. |
| 49 | + |
| 50 | +> To some degree, this is inevitable as NVIDIA's hardware architecture contains proprietary information that is not publicly available if you want to fully understand what's going on. However, there are still many things you can learn about how GPUs work in general, and it is a very useful skill to have if you want to go deeper into the engineering side of machine learning. |
| 51 | +
|
| 52 | +In my opinion, this leads to a severe degradation in quality of open sourced research code as _some_ researchers become used to spamming high level python code without much thought into the engineering aspect of their code, producing code that definitely would not have passed code review in a sane minded engineering organization. But that's just a side rant, and you get the point: this is a very common example of a black box that many people do not put too much emphasis on understanding. And it is true that you can treat the GPU as a black box most of the time here: PyTorch is simply too good. |
| 53 | + |
| 54 | +### Competitive programming - Network Flow |
| 55 | + |
| 56 | +One of the more complex concepts that is often required in competitive programming is [network flow](https://en.wikipedia.org/wiki/Network_flow_problem) and its family of algorithms. Most commonly used algorithms are probably the following(in order of increasing complexity): |
| 57 | + |
| 58 | +- Ford-Fulkerson method |
| 59 | + - Easy but inefficient |
| 60 | +- Edmonds-Karp algorithm |
| 61 | + - Somewhat understandable and efficient enough for many problems |
| 62 | +- Dinic's algorithm |
| 63 | + - More complex but very efficient, perhaps the point where you want to stop fully understanding the algorithm before using it |
| 64 | +- Push-relabel algorithm |
| 65 | + - Even more complex but more efficient, probably the efficiency limit of competitive programming due to the increasing implementation complexity |
| 66 | + |
| 67 | +The concept of Network Flow itself is already quite foreign to many competitive programmers who have just started to learn this subject, and many have a hard time understanding the basics up to the Edmonds-Karp algorithm. Just like in other subjects often touched in competitive programming, one may naturally expect that you must fully understand how the algorithm works in order to be able to slightly tweak them in favorable ways to solve advanced problems. This often happens for many other algorithms such as [Segment Tree Beats](https://usaco.guide/adv/segtree-beats?lang=cpp), [Square Root Decomposition](https://cp-algorithms.com/data_structures/sqrt_decomposition.html), [Sprague-Grundy theorem](https://cp-algorithms.com/game_theory/sprague-grundy-nim.html) and many more. |
| 68 | + |
| 69 | +However, starting from Dinic's algorithm, both the implementation complexity and the mathematical complexity of the algorithm increases to a point where competitive programmers without a mathematical background may find it very hard to fully understand why the algorithm works in said time complexity. There are numerous cases for Dinic's algorithm where it runs surprisingly fast on certain special graph variants, which is sometimes required for the programmer to tweak the input graph into such special variants so that the problem can be solved under the given time constraints. _Understanding why it becomes so fast_ often is much harder than merely understanding the base algorithm itself, and many competitive programmers often give up on fully understanding the algorithm at this point, and just treat it as a black box that can be used to solve problems. |
| 70 | + |
| 71 | +The reason why this is accepted as somewhat of a common practice is due to two reasons: |
| 72 | + |
| 73 | +- The key difficulty in network flow problems is often in modeling the problem into a flow network, that can utilize network flow algorithms to yield the desired results. |
| 74 | +- The harder a network flow problem gets, you usually end up _not being able to use network flow algorithms to solve them_ at all due to the massive input constraints. Network Flow at this level is often used as a stepping stone to transform the problem's graph structure into something else that can be more efficiently approached with something like a complex dynamic programming / greedy approach. |
| 75 | + - A common example of this may be identifying a [min-cost flow problem to utilize its convexity](https://codeforces.com/blog/entry/118391). |
| 76 | + |
| 77 | +Therefore in this case, leaving Dinic's algorithm and above as a black box is often an acceptable practice, and no teammate would ever question you if you were to simply copy-paste a library implementation of said algorithms in a team contest. |
| 78 | + |
| 79 | +# Preparing for opening the black box |
| 80 | + |
| 81 | +Ok, now we've come to accept that: |
| 82 | + |
| 83 | +- Black boxes are somewhat inevitable |
| 84 | +- Black boxes are actually useful to skip unnecessary complexity |
| 85 | + |
| 86 | +But now, we have to answer the first ever question of how we prepare ourselves for the situation where we actually need to open the black box. If we aren't going to be actively opening them, how do we prepare ourselves to understand such complex concepts when the time comes? |
| 87 | + |
| 88 | +The answer is very simple in most cases: **refine your foundations**. |
| 89 | + |
| 90 | +I'm going to be very honest here from what I've seen including my own experiences and say that building a strong foundation from scratch is pretty much impossible. There is almost no way you can start from zero in a certain subject and be able to confidently explain every fundamental concept that is recommended to beginners, unless you are one of those rare geniuses that are destined to become a prodigy in that field. The reason is simple: you don't know what you don't know. Realizing what you don't know takes a lot more experience / knowledge than you may think in the first place, and often times you will discover missing holes in your foundational knowledge as you progress further into the subject. This is a very natural process, and you should not be discouraged by this. |
| 91 | + |
| 92 | +However, this is also why many 'unnecessary' black boxes may arise. In the most optimal scenario, you would be an absolute expert in every fundamental concept, and thus be able to understand everything on the go as you learn something new. When this process _fails_, ideally you will be able to identify where the missing link is in terms of your foundational skills. |
| 93 | + |
| 94 | +- Perhaps you don't fully understand how the written math equation makes sense. You may realize that your weakness in linear algebra has finally caught up to you, and that its time to go back and relearn some of the basics. |
| 95 | + - Still happens to me all the time. I'm absolutely terrible at linear algebra, so this is an area that I'm constantly trying to improve on. |
| 96 | +- Perhaps you don't fully understand why the given code is so efficient in practice. You may recall from your past experiences that you've kind of skimmed through some advanced utilizations of simpler data structures, and that you need to go back and solidify your understanding of those simpler data structures first. |
| 97 | + - Happened to me a lot in competitive programming, especially with hard dynamic programming problems. I often found myself not fully understanding how one would come up with such magical looking recurrence relations and tricks, and it turned out that I was simply lacking experience in solving simpler DP problems that would have naturally given me the intuition to solve the harder ones. |
| 98 | + - Funnily enough DP is yet another common black box in RL. If you were to look at how deep DP can get, you may want to give a quick look at some ICPC WF problems that could make IGM+s scratch their heads for hours. Even that is probably scratching the surface of theoretical DP research. |
| 99 | + |
| 100 | +If you don't have such sparking realizations, it means the problem is worse than you had anticipated: You are likely lacking so much foundational knowledge required for the given problem that you don't even know where to start! A common cause in this case is that the required intuition stems from perhaps a seemingly unrelated field that you have never touched before. |
| 101 | + |
| 102 | +- Software Engineering is often overlooked by many researchers and scientists, but it is often an important skill when it comes to understanding and implementing efficient code. |
| 103 | +- Mathematics is always a common missing link for many programmers. When in doubt, its probably math. |
| 104 | +- There is a reason why every beginner book repeats the same thing over and over again: even if something may seem outdated or irrelevant to what you're learning today, the intuition gained from learning it is often useful in unexpected ways. |
| 105 | + |
| 106 | +If you encounter a black box that you don't really want to open at the moment, just make sure to keep yourself prepared for a hypothetical situation where you **need** to open it. Identify the missing links that have prevented you from fully understanding it at the moment, and make sure to keep track of them and perhaps follow up on them when you have some spare time. If you find yourself maybe experiencing a lot of black boxes caused from the lack of mathematical intuition, perhaps it is time to go learn some math. Maybe you shouldn't have skipped that one core CS course in university, so it's time to learn it on your own. Maybe you shouldn't have been using typeless python all the time that makes you write code that is horrendous to read/maintain, so it's time to learn a more strongly typed language such as Rust or C++. |
| 107 | + |
| 108 | +# The age of continuous learning |
| 109 | + |
| 110 | +Thankfully, LLMs in this regard are more helpful than ever when you want guidance to learning new topics. In my opinion, using such AI tools to _aid_ your work rather than completely fulfilling them is extremely useful to grow your skills in previously unconfident directions. Once you start to identify your weak spots, the act of learning really never ends. The more you unveil that you do not know yet, the more you realize how much skill you are lacking. Honestly, it's to the degree where I sometimes feel like people _not_ suffering from imposter syndrome are ones who are simply turning a blind eye to their own weaknesses. |
| 111 | + |
| 112 | +Opportunities may seem rarer compared to the past, with standards also rising higher than ever, but the amount of information available to us is also unprecedentedly high. The amount of things you can achieve on your own as a software engineer is almost absurdly high compared to even a few years ago, the pre-LLM era. Learn to keep yourself away from slop, but don't make that deter you from utilizing the tools available to you. The world is moving faster than ever, and the only way to keep up is to keep learning. |
0 commit comments