ilusha/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython synced 2026-09-29 08:45:29 +03:00
Code Issues Projects Releases Packages Wiki Activity Actions
cpython/Include
History
Ivan Levkivskyi 9932a22897
bpo-33416: Add end positions to Python AST (GH-11605) 
The majority of this PR is tediously passing `end_lineno` and `end_col_offset` everywhere. Here are non-trivial points:
* It is not possible to reconstruct end positions in AST "on the fly", some information is lost after an AST node is constructed, so we need two more attributes for every AST node `end_lineno` and `end_col_offset`.
* I add end position information to both CST and AST.  Although it may be technically possible to avoid adding end positions to CST, the code becomes more cumbersome and less efficient.
* Since the end position is not known for non-leaf CST nodes while the next token is added, this requires a bit of extra care (see `_PyNode_FinalizeEndPos`). Unless I made some mistake, the algorithm should be linear.
* For statements, I "trim" the end position of suites to not include the terminal newlines and dedent (this seems to be what people would expect), for example in
  ```python
  class C:
      pass

  pass
  ```
  the end line and end column for the class definition is (2, 8).
* For `end_col_offset` I use the common Python convention for indexing, for example for `pass` the `end_col_offset` is 4 (not 3), so that `[0:4]` gives one the source code that corresponds to the node.
* I added a helper function `ast.get_source_segment()`, to get source text segment corresponding to a given AST node. It is also useful for testing.

An (inevitable) downside of this PR is that AST now takes almost 25% more memory. I think however it is probably justified by the benefits.
2019-01-22 11:18:22 +00:00
..
cpython bpo-35444: Unify and optimize the helper for getting a builtin object. (GH-11047) 2018-12-11 08:28:18 +02:00
internal bpo-35758: Fix building on ARM + MSVC (gh-11531) 2019-01-21 21:49:40 +01:00
abstract.h bpo-35134: Create Include/cpython/abstract.h (GH-10728) 2018-11-26 22:42:04 +01:00
asdl.h bpo-18407: ast.c uses Py_ssize_t for asdl_seq_LEN() iterator (GH-10655) 2018-11-22 14:45:16 +01:00
ast.h bpo-35177: Add dependencies between header files (GH-10361) 2018-11-12 00:56:19 +01:00
bitset.h bpo-32150: Expand tabs to spaces in C files. (#4583) 2017-11-28 17:56:10 +02:00
bltinmodule.h
boolobject.h
bytearrayobject.h
bytes_methods.h bpo-32677: Add .isascii() to str, bytes and bytearray (GH-5342) 2018-01-27 14:06:21 +09:00
bytesobject.h bpo-32150: Expand tabs to spaces in C files. (#4583) 2017-11-28 17:56:10 +02:00
cellobject.h bpo-32150: Expand tabs to spaces in C files. (#4583) 2017-11-28 17:56:10 +02:00
ceval.h bpo-35444: Unify and optimize the helper for getting a builtin object. (GH-11047) 2018-12-11 08:28:18 +02:00
classobject.h bpo-32150: Expand tabs to spaces in C files. (#4583) 2017-11-28 17:56:10 +02:00
code.h String annotations [PEP 563] (#4390) 2018-01-26 08:20:18 -08:00
codecs.h
compile.h bpo-32455: Add jump parameter to dis.stack_effect(). (GH-6610) 2018-09-18 09:54:26 +03:00
complexobject.h
context.h bpo-34762: Fix contextvars C API to use PyObject* pointer types. (GH-9473) 2018-09-21 15:33:56 -04:00
coreconfig.h bpo-35233: InitConfigTests tests more config vars (GH-10541) 2018-11-14 17:39:45 +01:00
datetime.h bpo-35081: Remove Py_BUILD_CORE from datetime.h (GH-10416) 2018-11-13 09:02:25 +01:00
descrobject.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
dictobject.h bpo-35134: Create Include/cpython/dictobject.h (GH-10732) 2018-11-27 00:12:26 +01:00
dtoa.h
dynamic_annotations.h
enumobject.h
errcode.h bpo-32150: Expand tabs to spaces in C files. (#4583) 2017-11-28 17:56:10 +02:00
eval.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
fileobject.h bpo-29240: PEP 540: Add a new UTF-8 Mode (#855) 2017-12-13 12:29:09 +01:00
fileutils.h bpo-28604: Fix localeconv() for different LC_MONETARY (GH-10606) 2018-11-20 16:20:16 +01:00
floatobject.h
frameobject.h Move exc state to generator. Fixes bpo-25612 (#1773) 2017-10-22 23:41:51 +02:00
funcobject.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
genobject.h bpo-32591: Add native coroutine origin tracking (#5250) 2018-01-21 09:44:07 -05:00
graminit.h bpo-30406: Make async and await proper keywords (#1669) 2017-10-05 23:24:46 -04:00
grammar.h bpo-32150: Expand tabs to spaces in C files. (#4583) 2017-11-28 17:56:10 +02:00
import.h Fix redundant declaration of _PyImport_AddModuleObject (GH-7992) 2018-06-29 16:47:18 -06:00
intrcheck.h Add missed details of the C API introduced in 3.7. (GH-7047) 2018-05-22 20:59:42 +03:00
iterobject.h
listobject.h
longintrepr.h bpo-32150: Expand tabs to spaces in C files. (#4583) 2017-11-28 17:56:10 +02:00
longobject.h bpo-20260: Implement non-bitwise unsigned int converters for Argument Clinic. (GH-8434) 2018-07-26 13:22:16 +03:00
marshal.h
memoryobject.h
metagrammar.h
methodobject.h bpo-32240: Add the const qualifier to declarations of PyObject* array arguments. (#4746) 2017-12-15 13:11:11 +02:00
modsupport.h bpo-35582: Argument Clinic: inline parsing code for positional parameters. (GH-11313) 2019-01-11 16:01:14 +02:00
moduleobject.h bpo-33237: Improve AttributeError message for partially initialized module. (GH-6398) 2018-10-30 13:19:51 +02:00
namespaceobject.h
node.h bpo-33416: Add end positions to Python AST (GH-11605) 2019-01-22 11:18:22 +00:00
object.h bpo-35134: Don't define types twice in header files (GH-10754) 2018-11-27 23:54:59 +01:00
objimpl.h bpo-35134: Create Include/cpython/ subdirectory (GH-10624) 2018-11-23 17:00:00 +01:00
odictobject.h bpo-35059: Add _PyObject_CAST() macro (GH-10645) 2018-11-22 02:57:29 +01:00
opcode.h Update opcode.h header comment to mention the source data file (GH-9935) 2018-10-18 14:53:18 -05:00
osdefs.h
osmodule.h
parsetok.h bpo-35177: Add dependencies between header files (GH-10361) 2018-11-12 00:56:19 +01:00
patchlevel.h Start of 3.8.0a0 2018-01-31 17:44:09 -05:00
pgen.h
pgenheaders.h bpo-32150: Expand tabs to spaces in C files. (#4583) 2017-11-28 17:56:10 +02:00
py_curses.h bpo-20171: Convert the _curses and _curses_panel modules to Argument Clinic. (GH-4251) 2018-05-10 11:27:23 +03:00
pyarena.h
pycapsule.h
pyctype.h
pydebug.h bpo-32125: Remove Py_UseClassExceptionsFlag flag (#4544) 2017-11-24 22:06:38 +01:00
pydtrace.d bpo-31574: importlib dtrace (#3749) 2017-09-29 15:53:19 -07:00
pydtrace.h bpo-35059: Remove Py_STATIC_INLINE() macro (GH-10216) 2018-10-29 14:49:24 +01:00
pyerrors.h bpo-35134: Add Include/cpython/pyerrors.h (GH-10727) 2018-11-26 22:11:25 +01:00
pyexpat.h bpo-34623: Use XML_SetHashSalt in _elementtree (GH-9146) 2018-09-18 05:38:58 -07:00
pyfpe.h bpo-29137: Remove fpectl module (#4789) 2018-01-05 23:15:34 -08:00
pyhash.h Correct the location of a function mentioned in a comment (GH-4327) 2017-11-09 19:55:34 +03:00
pylifecycle.h bpo-35259: Limit Py_FinalizeEx() to Py_LIMITED_API >= 0x03060000. (GH-10620) 2018-12-20 23:11:03 +02:00
pymacconfig.h
pymacro.h bpo-31338 (#3374) 2017-09-14 18:13:16 -07:00
pymath.h bpo-31373: fix undefined floating-point demotions (#3396) 2017-09-07 11:13:59 -07:00
pymem.h bpo-35081: Move Py_BUILD_CORE code to internal/mem.h (GH-10249) 2018-10-31 20:19:24 +01:00
pyport.h bpo-35758: Fix building on ARM + MSVC (gh-11531) 2019-01-21 21:49:40 +01:00
pystate.h bpo-35134: Don't define types twice in header files (GH-10754) 2018-11-27 23:54:59 +01:00
pystrcmp.h
pystrhex.h
pystrtod.h
Python-ast.h bpo-33416: Add end positions to Python AST (GH-11605) 2019-01-22 11:18:22 +00:00
Python.h bpo-28503: Use crypt_r() when available instead of crypt() (GH-11373) 2018-12-30 15:42:32 -08:00
pythonrun.h
pythread.h bpo-30768: Recompute timeout on interrupted lock (GH-4103) 2017-10-24 16:53:32 -07:00
pytime.h bpo-31784: Implement PEP 564: add time.time_ns() (#3989) 2017-11-02 07:28:27 -07:00
rangeobject.h
setobject.h
sliceobject.h bpo-32150: Expand tabs to spaces in C files. (#4583) 2017-11-28 17:56:10 +02:00
structmember.h
structseq.h
symtable.h bpo-35177: Add dependencies between header files (GH-10361) 2018-11-12 00:56:19 +01:00
sysmodule.h bpo-32030: Make _PySys_AddXOptionWithError() private (GH-10236) 2018-10-30 14:31:42 +01:00
token.h bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370) 2018-12-22 11:18:40 +02:00
traceback.h
tracemalloc.h bpo-35053: Define _PyTraceMalloc_NewReference in object.h (GH-10107) 2018-10-26 00:01:56 +02:00
tupleobject.h bpo-35134: Create Include/cpython/tupleobject.h (GH-10764) 2018-11-28 13:01:32 +01:00
typeslots.h
ucnhash.h
unicodeobject.h bpo-35134: Create Include/cpython/unicodeobject.h (GH-10680) 2018-11-26 17:29:38 +01:00
warnings.h bpo-32591: Add native coroutine origin tracking (#5250) 2018-01-21 09:44:07 -05:00
weakrefobject.h