44398d29ea
The new lexer matches the TOML spec much more closely. User-visible differences should be these: * Add MIME type * Highlight string escapes * Recognize \uXXXX and \UXXXX escapes * Also recognize booleans if they are followed by a comment * Fix single quotes inside multiline literal strings (closes #2488) * Prevent multiline literal strings from eating comments * Add multiline basic strings (""") * Improve datetime recognition: recognize times without dates, dates without times and datetimes without time zone; allow sub-millisecond precision * Recognize floats with exponents (they used not to be recognized when having a decimal point) * Recognize binary, octal and hex literals * Recognize strings inside table headers * Recognize table headers followed by comments * Don't parse sequences of digits as integers when they are actually keys Includes several new tests, most of which were not working before.
27 lines
731 B
Text
27 lines
731 B
Text
---input---
|
|
[table.header.with.some."strings".and."escape\nsequences\u263a"]
|
|
|
|
---tokens---
|
|
'[' Keyword
|
|
'table' Keyword
|
|
'.' Keyword
|
|
'header' Keyword
|
|
'.' Keyword
|
|
'with' Keyword
|
|
'.' Keyword
|
|
'some' Keyword
|
|
'.' Keyword
|
|
'"' Literal.String.Double
|
|
'strings' Literal.String.Double
|
|
'"' Literal.String.Double
|
|
'.' Keyword
|
|
'and' Keyword
|
|
'.' Keyword
|
|
'"' Literal.String.Double
|
|
'escape' Literal.String.Double
|
|
'\\n' Literal.String.Escape
|
|
'sequences' Literal.String.Double
|
|
'\\u263a' Literal.String.Escape
|
|
'"' Literal.String.Double
|
|
']' Keyword
|
|
'\n' Text.Whitespace
|