Upcoming Plans
Starting a series
Whether it lasts depends entirely on my motivation, but for a while I’m going to write about the fundamentals of High Performance Computing, aiming for roughly one post a week. (That’s exactly why I picked Jekyll, so I could keep the blog going just by writing Markdown.)
By “fundamentals” I don’t mean “easy”; I mean the fundamental, basic stuff. The idea is to write, based on my own experience, about the walls I hit from joining a lab as a fourth-year undergrad through the second year of my master’s, and the mistakes I made because I was missing some basic way of thinking.
Why do this? Compared to web performance tuning like ISUCON, High Performance Computing doesn’t have many books or articles that cover it systematically. It feels like much of it is built on tacit knowledge shared among people in Japan who already know it, so I thought I’d try to write some of it down. That said, this is just the blog of a master’s student, so I’ll possibly write wrong things.
Another reason is that I want to write about CUDA Extended GPU Memory on this blog, and I want to first provide the background knowledge readers should have for that. Also, every time a new device or model comes out, I see a lot of people who just say “it’s so fast, it’s amazing,” so I’m hoping to see more discussion in Japan about why new devices enables the program much faster.
I’m assuming readers have the background of someone who has studied for four years in a computer science department. If you’ve taken courses on programming languages, operating systems, and basic algorithms, you should be fine.
Contents
I’ll list topics in the order they occur to me while writing. So far I’ve come up with two.
1. How to read experimental data and spec sheets
Next week I’ll mainly write about what you should think about when looking at experimental data. This is necessary for judging whether your measurements are actually valid. Using matrix multiplication as the example, the post will cover how to check whether a measured value really makes sense by comparing it against the catalog specs in the datasheet. After reading it, you’ll no longer be fooled by hype like “new-generation architecture” or “next-generation processor.”
Keywords: Bandwidth, FLOPS, and Roofline.
2. Modeling, hypotheses, and estimation
In “1. How to read experimental data and spec sheets,” we checked the validity of measurements for a single operation. In practice, though, unless you’re a CUDA kernel or SIMD tuning specialist, you rarely look at just one kernel; you’re trying to speed up the whole system. I’ll use concrete examples to explain how to approach that situation. I’ll also explain how to estimate lower and upper bounds for measured values.
Keywords: Profile, modeling.
After covering these two topics, I’ll move on to explaining CUDA Extended GPU Memory and LLM Serving.