To recap: it’s November 2025 and I’ve spent the last two-ish months working on identifier interning. My plans to implement name resolution by the end of the year were screwed, but I still wanted to spend substantial time on it.
To be honest, I think my foray into identifier interning was really to avoid implementing parsing. Parsing is just really, really hard. The Rust syntax is deeply complicated, it varies (in subtle ways) across editions, and you need to design an AST to parse into. I had already put a lot of effort into designing my own token stream format, which meant I couldn’t use existing parsers.
Hearkening back to my old post about making an AST, I spent a few weeks trying to implement the AST I had envisioned. I laid out the different sorts of AST nodes (expressions, statements, patterns, types, etc.); I even wrote most of the code. But the design didn’t feel right.
The core idea of my AST design was bottom-up linking; each node in the tree would point to its parent instead of its children. I believed this would unlock interesting and efficient traversal patterns. The biggest problem with this design was separating node sorts; expressions and patterns are different sorts of AST nodes and should be stored separately, but they sometimes contain each other. This is much harder to express with bottom-up linking than with regular top-down linking.
The bigger problem, though, was that the concept of an AST didn’t feel right. The AST may be useful for some lints, but it’s not directly relevant to the hot path of compilation. An AST’s primary purpose is to be lowered into the HIR (high-level intermediate representation, where type-checking usually occurs). So why create it at all? While struggling to overcome the difficulties of the AST I had put together, I was questioning whether it was necessary at all — I could imagine a streaming parser would be far more useful.
After a few days of agonizing over the problem, I recognized that I would not have a good answer quickly, but I still wanted to implement name resolution quickly. Eventually, I decided to disable my lexer entirely and just use syn to get a simple (imperfect) AST that I could work with.
As of today, this hack is still present in the codebase. I think a streaming parser would be a few months of effort, and I’d gladly take it on once a basic name resolution implementation is complete.