Grammars, Syntax, Semantics, and AST

1. Levels of Languages: Regex (L3), Context-Free (L2), Context-Sensitive (L1), Recursively Enumerable (L0).

1. Defined using Terminals (Tokens aka Lexemes), and Non-Terminals
2. Start Symbol (a non-terminal)
3. Non-terminals aka Syntactic Categories. Non-terminals are called that because a sentence production has not ended yet.
4. There is no "context"
   1. A grammar production rule: LHS ::= RHS
   2. LHS is a single non-terminal (i.e., without context)
   3. RHS is a seq of terminals and non-terminals

1. Limitations of CFG
   1. Defines syntax upto a "level"
   2. Cannot capture "context"
      1. E.g., "variable should be declared before use"
2. CFGs do not define semantics
   1. minor exceptions exist
3. Every modern PL has a CFG, often several.

1. The production rules generate a sequence of tokens. If the production is starting from the start non-terminal, a sentence in the language is generated.
2. A rendering of a sequence of tokens as a string of characters will (usually) separate the tokens by non-empty white spaces.
3. Sometimes the lexical structure is specified using CFGs. Here, the terminals are characters, and there is no white space separation.

1. Simplify the BNF notation.
2. Drop <> from non-terminals.
3. Show terminals in a different font, or quote them.
4. Use {} or * for repetitions.
5. See the Java Grammar (below).

1. Syntax of most languages is "specified" using context-free grammars.
2. Almost always incompletely (because we use context-free grammars).
3. Almost never ambiguously (unique derivation trees, described below). The word ambiguous is not for semantics, but for parsing.

1. Chapter2. Grammars [from Oracle Java] "This chapter describes the context-free grammars used in this specification to define the lexical and syntactic structure of a program."
2. Chapter18. Java [from Oracle Java] "This chapter presents a grammar for the Java programming language."
3. https://kotlinlang.org/docs/reference/grammar.html Kotlin grammar

1. http://www.open-std.org/JTC1/SC22/WG14/ C
2. http://www.open-std.org/JTC1/SC22/WG21/ C++
3. Standard for Programming Language C++, Working Draft, 1600+ pp. 2018.

1. A Grammar and an Example Derivation Tree. The grammar has just one production rule.

<assign> -> <id> := <expr>
<id> -> A | B | C
<expr> -> <expr> + <expr>
  | <expr> * <expr>
  | ( <expr> )
  | <id>

1. Derivation tree of x + 3 * y
2. From https://www.student.cs.uwaterloo.ca/~cs241/cfg/cfg.html ;;
3. In the above CFG,
   1. ::= and <> are omitted
   2. id, +, *, # are lexemes

1. An AST example of x + 3 * y

     +
    / \
   x   *
      / \
     3   y
   

1. Examples of AST drawings often do not show symbol-tables, but they are included.
2. In the above AST: a, b were positive integers initialized by the caller.
3. Exercise: Deduce the source code from the above AST