44398d29ea
The new lexer matches the TOML spec much more closely. User-visible differences should be these: * Add MIME type * Highlight string escapes * Recognize \uXXXX and \UXXXX escapes * Also recognize booleans if they are followed by a comment * Fix single quotes inside multiline literal strings (closes #2488) * Prevent multiline literal strings from eating comments * Add multiline basic strings (""") * Improve datetime recognition: recognize times without dates, dates without times and datetimes without time zone; allow sub-millisecond precision * Recognize floats with exponents (they used not to be recognized when having a decimal point) * Recognize binary, octal and hex literals * Recognize strings inside table headers * Recognize table headers followed by comments * Don't parse sequences of digits as integers when they are actually keys Includes several new tests, most of which were not working before.
64 lines
1.7 KiB
Text
64 lines
1.7 KiB
Text
---input---
|
|
1234 = "foo" # 1234 is a name
|
|
foo = 1234 # 1234 is a number
|
|
foo2 = [
|
|
1234, # number
|
|
{ 1234 = "foo", foo = 1234 } # name then number
|
|
]
|
|
|
|
---tokens---
|
|
'1234' Name
|
|
' ' Text.Whitespace
|
|
'=' Operator
|
|
' ' Text.Whitespace
|
|
'"' Literal.String.Double
|
|
'foo' Literal.String.Double
|
|
'"' Literal.String.Double
|
|
' ' Text.Whitespace
|
|
'# 1234 is a name' Comment.Single
|
|
'\n' Text.Whitespace
|
|
|
|
'foo' Name
|
|
' ' Text.Whitespace
|
|
'=' Operator
|
|
' ' Text.Whitespace
|
|
'1234' Literal.Number.Integer
|
|
' ' Text.Whitespace
|
|
'# 1234 is a number' Comment.Single
|
|
'\n' Text.Whitespace
|
|
|
|
'foo2' Name
|
|
' ' Text.Whitespace
|
|
'=' Operator
|
|
' ' Text.Whitespace
|
|
'[' Punctuation
|
|
'\n ' Text.Whitespace
|
|
'1234' Literal.Number.Integer
|
|
',' Punctuation
|
|
' ' Text.Whitespace
|
|
'# number' Comment.Single
|
|
'\n ' Text.Whitespace
|
|
'{' Punctuation
|
|
' ' Text.Whitespace
|
|
'1234' Name
|
|
' ' Text.Whitespace
|
|
'=' Punctuation
|
|
' ' Text.Whitespace
|
|
'"' Literal.String.Double
|
|
'foo' Literal.String.Double
|
|
'"' Literal.String.Double
|
|
',' Punctuation
|
|
' ' Text.Whitespace
|
|
'foo' Name
|
|
' ' Text.Whitespace
|
|
'=' Punctuation
|
|
' ' Text.Whitespace
|
|
'1234' Literal.Number.Integer
|
|
' ' Text.Whitespace
|
|
'}' Punctuation
|
|
' ' Text.Whitespace
|
|
'# name then number' Comment.Single
|
|
'\n' Text.Whitespace
|
|
|
|
']' Punctuation
|
|
'\n' Text.Whitespace
|