44398d29ea
The new lexer matches the TOML spec much more closely. User-visible differences should be these: * Add MIME type * Highlight string escapes * Recognize \uXXXX and \UXXXX escapes * Also recognize booleans if they are followed by a comment * Fix single quotes inside multiline literal strings (closes #2488) * Prevent multiline literal strings from eating comments * Add multiline basic strings (""") * Improve datetime recognition: recognize times without dates, dates without times and datetimes without time zone; allow sub-millisecond precision * Recognize floats with exponents (they used not to be recognized when having a decimal point) * Recognize binary, octal and hex literals * Recognize strings inside table headers * Recognize table headers followed by comments * Don't parse sequences of digits as integers when they are actually keys Includes several new tests, most of which were not working before.
15 lines
439 B
Text
15 lines
439 B
Text
---input---
|
|
message = """multiline strings
|
|
can be followed by""" # comments
|
|
|
|
---tokens---
|
|
'message' Name
|
|
' ' Text.Whitespace
|
|
'=' Operator
|
|
' ' Text.Whitespace
|
|
'"""' Literal.String.Double
|
|
'multiline strings\n can be followed by' Literal.String.Double
|
|
'"""' Literal.String.Double
|
|
' ' Text.Whitespace
|
|
'# comments' Comment.Single
|
|
'\n' Text.Whitespace
|