I’m ecstatic to announce that, from July 22, I will be paid to work on rustc and Krabby for one day a week. This is an internship at RustNL (as part of The Rust Maintainers Team) enabled by my employer NLnet Labs, which will last up to a year. I’m immensely grateful to everyone involved in making this happen! In this post, I’m going to lay out how I will spend this time.
When I had started working on Krabby, my plan had been to develop it entirely independently from rustc, to build a vision of what rustc could look like if it was optimized in every way I could think of. This layout gave me a lot of freedom to approach the architecture however I wanted, and made the overall project more achievable: because Krabby would be fully independent of rustc, I could avoid supporting all the edge cases rustc considers, significantly reducing my workload (even though I think most edge cases would fit within the architecture I design, they would require a lot of effort to implement).
But as I worked on Krabby over the last year, my views changed. One of my biggest sources of motivation is helping improve compilation performance for everyone. Contributing optimizations to rustc has immediate positive impacts, even though it can be more time-consuming than building the equivalent code in Krabby (since rustc has a higher bar for quality, e.g. strong stability requirements). By looking at rustc’s code, I can gain a better understanding of compiler features and think of more ways to optimize them.
After my post about a degenerate case in macro parsing, I took a deep dive into rustc’s code for declarative macros, and I thought of several ways to optimize it. Over the last month, I’ve been contributing these optimizations: see for example #158974 and #158976. Now that I have a clear understanding of how this code works, I am confident I can copy it into Krabby and try out even bigger (but harder to contribute) architectural changes! I’m quite happy with this approach and I want to keep at it with the later stages of compilation too.
Meanwhile, I am also developing higher-level architectural changes for Krabby. I have prepared a design for Krabby’s query system, which I am in the middle of writing a blog post for. I hope to publish it soon; it will form the basis for my EuroRust talk. I plan to implement it over the next few months and then refactor Krabby’s existing code to use it. I plan to continue doing this high-level work alongside the development of specific components like macro expansion.
One aspect of optimization that I find really critical is data collection. While optimizing rustc’s macro parsing, I found myself asking “what’s the average length of a macro rule? what is the frequency distribution of lengths?” As far as I can tell, the Rust project does not have infrastructure for measuring such properties easily. I want to extract every single macro invocation performed by rustc to compile the top 10,000 crates on https://crates.io and I want them in a single plain text file for analysis. Collecting and sharing this data should help us find new ways to optimize the code; I want to promote this paradigm and help rustc adopt infrastructure to make this easier.
So, what’s the overall plan? Over the duration of the internship, I plan to work on Krabby and rustc through these three approaches. I will dive into a particular component of rustc, collect data about real-world inputs, contribute optimizations back to rustc, copy the code into Krabby, and explore large-scale optimizations. I plan to follow this approach for macro expansion (ongoing), parsing, and name resolution. Outside specific components, I will work on Krabby’s high-level architecture, especially its query system, partly in preparation for my EuroRust talk.
I’m going to post periodic updates about the internship and what I’ve achieved, both within rustc and within Krabby. Also, if you’ve been here before, you might notice that I’ve updated the top-level page about Krabby to reflect my new approach. If you’re interested in Krabby and want to help, don’t be afraid to reach out! You can find me on Zulip or send me an e-mail. I need help implementing things in Krabby and adding infrastructure (e.g. benchmarks, tests, fuzzing, data collection). If you have wild ideas for making Rust compilation faster, I’m always happy to chat :D
I’ve been working on Krabby in my free time until now, and while it was enough to get the project off the ground, this internship will give me the time I need to be more productive and to work more comfortably. I’m so excited to get started, and once again, I’m incredibly thankful to get this opportunity.