ilusha/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython synced 2026-09-29 12:10:30 +03:00
Code Issues Projects Releases Packages Wiki Activity Actions

bpo-40334: Catch E_EOF error, when the tokenizer returns ERRORTOKEN (GH-19743)

Browse source 
An E_EOF error was only being caught after the parser exited before this commit. There are some cases though, where the tokenizer returns ERRORTOKEN *and* has set an E_EOF error (like when EOF directly follows a line continuation character) which weren't correctly handled before.
This commit is contained in:
Lysandros Nikolaou 2020-04-28 03:23:35 +03:00 • committed by GitHub
parent 5d1f32d33b
commit d55133f49f
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
2 changed files with 6 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

9
Parser/pegen/pegen.c
View file

@ -344,13 +344,16 @@ tokenizer_error(Parser *p)
break;
case E_BADPREFIX:
return tokenizer_error_with_col_offset(p,
PyExc_SyntaxError, "invalid string prefix");
errtype, "invalid string prefix");
case E_EOFS:
return tokenizer_error_with_col_offset(p,
PyExc_SyntaxError, "EOF while scanning triple-quoted string literal");
errtype, "EOF while scanning triple-quoted string literal");
case E_EOLS:
return tokenizer_error_with_col_offset(p,
PyExc_SyntaxError, "EOL while scanning string literal");
errtype, "EOL while scanning string literal");
case E_EOF:
return tokenizer_error_with_col_offset(p,
errtype, "unexpected EOF while parsing");
case E_DEDENT:
return tokenizer_error_with_col_offset(p,
PyExc_IndentationError, "unindent does not match any outer indentation level");