44398d29ea
The new lexer matches the TOML spec much more closely. User-visible differences should be these: * Add MIME type * Highlight string escapes * Recognize \uXXXX and \UXXXX escapes * Also recognize booleans if they are followed by a comment * Fix single quotes inside multiline literal strings (closes #2488) * Prevent multiline literal strings from eating comments * Add multiline basic strings (""") * Improve datetime recognition: recognize times without dates, dates without times and datetimes without time zone; allow sub-millisecond precision * Recognize floats with exponents (they used not to be recognized when having a decimal point) * Recognize binary, octal and hex literals * Recognize strings inside table headers * Recognize table headers followed by comments * Don't parse sequences of digits as integers when they are actually keys Includes several new tests, most of which were not working before.
86 lines
3 KiB
Text
86 lines
3 KiB
Text
---input---
|
|
[strings]
|
|
basic-string = "I'm a basic string. I can contain 'single quotes', \u0055nicode escapes \U0001f61b \U0001F61B, \"escaped\" double quotes, \n and \t more."
|
|
literal-string = 'I am literal string. Escapes like \this have no effect on me. I can contain "double quotes".'
|
|
multiline-basic-string = """
|
|
I'm a multiline basic string.
|
|
I can span several lines and contain 'single' and "double" quotes
|
|
as well as \u0055nicode escapes. Line continuations \
|
|
work too.
|
|
"""
|
|
multiline-literal-string = '''
|
|
I'm a "multiline" 'literal' string.
|
|
Escapes like \this have no effect on me. Neither does this: \
|
|
it is not a line continuation.'''
|
|
|
|
---tokens---
|
|
'[' Keyword
|
|
'strings' Keyword
|
|
']' Keyword
|
|
'\n' Text.Whitespace
|
|
|
|
'basic-string' Name
|
|
' ' Text.Whitespace
|
|
'=' Operator
|
|
' ' Text.Whitespace
|
|
'"' Literal.String.Double
|
|
"I'm a basic string. I can contain 'single quotes', " Literal.String.Double
|
|
'\\u0055' Literal.String.Escape
|
|
'nicode escapes ' Literal.String.Double
|
|
'\\U0001f61b' Literal.String.Escape
|
|
' ' Literal.String.Double
|
|
'\\U0001F61B' Literal.String.Escape
|
|
', ' Literal.String.Double
|
|
'\\"' Literal.String.Escape
|
|
'escaped' Literal.String.Double
|
|
'\\"' Literal.String.Escape
|
|
' double quotes, ' Literal.String.Double
|
|
'\\n' Literal.String.Escape
|
|
' and ' Literal.String.Double
|
|
'\\t' Literal.String.Escape
|
|
' more.' Literal.String.Double
|
|
'"' Literal.String.Double
|
|
'\n' Text.Whitespace
|
|
|
|
'literal-string' Name
|
|
' ' Text.Whitespace
|
|
'=' Operator
|
|
' ' Text.Whitespace
|
|
"'" Literal.String.Single
|
|
'I am literal string. Escapes like \\this have no effect on me. I can contain "double quotes".\'' Literal.String.Single
|
|
'\n' Text.Whitespace
|
|
|
|
'multiline-basic-string' Name
|
|
' ' Text.Whitespace
|
|
'=' Operator
|
|
' ' Text.Whitespace
|
|
'"""' Literal.String.Double
|
|
"\nI'm a multiline basic string.\nI can span several lines and contain 'single' and " Literal.String.Double
|
|
'"' Literal.String.Double
|
|
'double' Literal.String.Double
|
|
'"' Literal.String.Double
|
|
' quotes\nas well as ' Literal.String.Double
|
|
'\\u0055' Literal.String.Escape
|
|
'nicode escapes. Line continuations ' Literal.String.Double
|
|
'\\' Literal.String.Escape
|
|
'\n' Text.Whitespace
|
|
|
|
' work too.\n' Literal.String.Double
|
|
|
|
'"""' Literal.String.Double
|
|
'\n' Text.Whitespace
|
|
|
|
'multiline-literal-string' Name
|
|
' ' Text.Whitespace
|
|
'=' Operator
|
|
' ' Text.Whitespace
|
|
"'''" Literal.String.Single
|
|
'\nI' Literal.String.Single
|
|
"'" Literal.String.Single
|
|
'm a "multiline" ' Literal.String.Single
|
|
"'" Literal.String.Single
|
|
'literal' Literal.String.Single
|
|
"'" Literal.String.Single
|
|
' string.\nEscapes like \\this have no effect on me. Neither does this: \\\n it is not a line continuation.' Literal.String.Single
|
|
"'''" Literal.String.Single
|
|
'\n' Text.Whitespace
|