HN Reader

This was a really great read! I'm wrote the tree sitter grammar for the Unison programming language, and discovered I really like the work involved in pattern matching that writing tokenizers and parsers comes down to. It also gives you an in-depth understanding of how the language works that you've writing a parser for, and how tooling works.

Like if you have an AST with the ability to map onto code that is displayed in your IDE, the algorithm for an IDE to refactor a variable name is to traverse up the AST until you get to the variable's declaration and then traverse all sibling trees, changing each matching name, but stopping a traversal whenever you encounter a new binding with same name. Code folding is to identify the categories of node that are "foldable" and then you hide every child of that node. Etc. It's all tree traversal algorithms.

It gives you a deep appreciation for how powerful the tooling can be thanks to proper parsing.

I think this makes it sound a lot more difficult than it has to be, with the formal theory.

When it's really one of the most simple things if you divide it in parts and look at it from a tokenizer (string to list of tokens) and parser on top. Where the tokenizer can usually be very simple: a loop, large switch on the current character, where a choice is made on "what can this be", and making it into a formal token or error. Then a simple recursive parser that can almost be a 1 to 1 copy of the (E)BNF.

I somewhat regularly stop to marvel that one of the greatest anarchist thinkers of our time is also responsible for foundational theories in linguistics that also correspond intimately with the foundational theories of computing. God bless Noam.

Very long write-up! If you read it and like the content, you might consider ditching Python and start using Raku a lot and often.

> quadruple G=(N,Σ,P,S), where T is a finite set of nonterminal symbols, Σ a finite set of terminal symbols, P is a set of production rules and S is a start symbol.

I think "T" is supposed to be "N" in that sentence[1], based solely upon the further use of "N" nomenclature in subsequent paragraphs

1: he said, 5 years too late into a forum just discussing the article

Theory without examples is like traveling without a memory.

I have experience with compilers, creating language from scratch, handwritten parsers

but language theory is always difficult to understand in its theoretical form

I've started with practice, so I think in terms of strings and operations on it (indices, substrings, loops, looks ahead, etc) and then I read this theory I strugle hard to understand it and understand why I'd want to use it

So, going with code examples make things easier for me

It's nice to review some of this theory after a week of coding my own interpreter. I have been studying about compilers at pikuma.com the whole week and reading this article after coding a parser is a great way of reviewing what I've implemented.

> All is set and done

For the record, the phrase is 'when all is said and done'[1].

[1]: https://en.wiktionary.org/wiki/when_all_is_said_and_done