44398d29ea
The new lexer matches the TOML spec much more closely. User-visible differences should be these: * Add MIME type * Highlight string escapes * Recognize \uXXXX and \UXXXX escapes * Also recognize booleans if they are followed by a comment * Fix single quotes inside multiline literal strings (closes #2488) * Prevent multiline literal strings from eating comments * Add multiline basic strings (""") * Improve datetime recognition: recognize times without dates, dates without times and datetimes without time zone; allow sub-millisecond precision * Recognize floats with exponents (they used not to be recognized when having a decimal point) * Recognize binary, octal and hex literals * Recognize strings inside table headers * Recognize table headers followed by comments * Don't parse sequences of digits as integers when they are actually keys Includes several new tests, most of which were not working before.
20 lines
552 B
Text
20 lines
552 B
Text
---input---
|
|
[example] # A comment can appear on the same line as a section header
|
|
foo = "bar"
|
|
|
|
---tokens---
|
|
'[' Keyword
|
|
'example' Keyword
|
|
']' Keyword
|
|
' ' Text.Whitespace
|
|
'# A comment can appear on the same line as a section header' Comment.Single
|
|
'\n' Text.Whitespace
|
|
|
|
'foo' Name
|
|
' ' Text.Whitespace
|
|
'=' Operator
|
|
' ' Text.Whitespace
|
|
'"' Literal.String.Double
|
|
'bar' Literal.String.Double
|
|
'"' Literal.String.Double
|
|
'\n' Text.Whitespace
|