Extract data from python stack frames and tracebacks for informative displays
6 | for i in range(5): 7 | row = [] 8 | result.append(row) --> 9 | print_stack() 10 | for j in range(5): ``` The code for `print_stack()` is fairly self-explanatory. If you want to learn more details about a particular class or method I suggest looking through some docstrings. `FrameInfo` is a class that accepts either a frame or a traceback object and provides a bunch of nice attributes and properties (which are cached so you don't need to worry about performance). In particular `frame_info.lines` is a list of `Line` objects. `line.render()` returns the source code of that line suitable for display. Without any arguments it simply strips any common leading indentation. Later on we'll see a more powerful use for it. You can see that `frame_info.lines` includes some lines of surrounding context. By default it includes 3 pieces of context before the main line and 1 piece after. We can configure the amount of context by passing options: ```python options = stack_data.Options(before=1, after=0) frame_info = stack_data.FrameInfo(frame, options) ``` Then the output looks like: ```
acquired package info from repodata DB: repodata/f1f85c23079e7764ad00378a9fc5d660b89a03657290e75426caf7be5b7da462-primary.sqlite.bz2
Get the currently executing AST node of a frame, and other information
[![Build Status](https://github.com/alexmojaki/executing/workflows/Tests/badge.svg?branch=master)](https://github.com/alexmojaki/executing/actions) [![Coverage Status](https://coveralls.io/repos/github/alexmojaki/executing/badge.svg?branch=master)](https://coveralls.io/github/alexmojaki/executing?branch=master) [![Supports Python versions 2.7 and 3.5+, including PyPy](https://img.shields.io/pypi/pyversions/executing.svg)](https://pypi.python.org/pypi/executing) This mini-package lets you get information about what a frame is currently doing, particularly the AST node being executed. * [Usage](#usage) * [Getting the AST node](#getting-the-ast-node) * [Getting the source code of the node](#getting-the-source-code-of-the-node) * [Getting the `__qualname__` of the current function](#getting-the-__qualname__-of-the-current-function) * [The Source class](#the-source-class) * [Installation](#installation) * [How does it work?](#how-does-it-work) * [Is it reliable?](#is-it-reliable) * [Which nodes can it identify?](#which-nodes-can-it-identify) * [Libraries that use this](#libraries-that-use-this) ```python import executing node = executing.Source.executing(frame).node ``` Then `node` will be an AST node (from the `ast` standard library module) or None if the node couldn't be identified (which may happen often and should always be checked). `node` will always be the same instance for multiple calls with frames at the same point of execution. If you have a traceback object, pass it directly to `Source.executing()` rather than the `tb_frame` attribute to get the correct node. For this you will need to separately install the [`asttokens`](https://github.com/gristlabs/asttokens) library, then obtain an `ASTTokens` object: ```python executing.Source.executing(frame).source.asttokens() ``` or: ```python executing.Source.for_frame(frame).asttokens() ``` or use one of the convenience methods: ```python executing.Source.executing(frame).text() executing.Source.executing(frame).text_range() ``` ```python executing.Source.executing(frame).code_qualname() ``` or: ```python executing.Source.for_frame(frame).code_qualname(frame.f_code) ``` Everything goes through the `Source` class. Only one instance of the class is created for each filename. Subclassing it to add more attributes on creation or methods is recommended. The classmethods such as `executing` will respect this. See the source code and docstrings for more detail. pip install executing If you don't like that you can just copy the file `executing.py`, there are no dependencies (but of course you won't get updates). Suppose the frame is executing this line: ```python self.foo(bar.x) ``` and in particular it's currently obtaining the attribute `self.foo`. Looking at the bytecode, specifically `frame.f_code.co_code[frame.f_lasti]`, we can tell that it's loading an attribute, but it's not obvious which one. We can narrow down the statement being executed using `frame.f_lineno` and find the two `ast.Attribute` nodes representing `self.foo` and `bar.x`. How do we find out which one it is, without recreating the entire compiler in Python? The trick is to modify the AST slightly for each candidate expression and observe the changes in the bytecode instructions. We change the AST to this: ```python (self.foo ** 'longuniqueconstant')(bar.x) ``` and compile it, and the bytecode will be almost the same but there will be two new instructions: LOAD_CONST 'longuniqueconstant' BINARY_POWER and just before that will be a `LOAD_ATTR` instruction corresponding to `self.foo`. Seeing that it's in the same position as the original instruction lets us know we've found our match. Yes - if it identifies a node, you can trust that it's identified the correct one. The tests are very thorough - in addition to unit tests which check various situations directly, there are property tests against a large number of files (see the filenames printed in [this build](https://travis-ci.org/alexmojaki/executing/jobs/557970457)) with real code. Specifically, for each file, the tests: 1. Identify as many nodes as possible from all the bytecode instructions in the file, and assert that they are all distinct 2. Find all the nodes that should be identifiable, and assert that they were indeed identified somewhere In other words, it shows that there is a one-to-one mapping between the nodes and the instructions that can be handled. This leaves very little room for a bug to creep in. Furthermore, `executing` checks that the instructions compiled from the modified AST exactly match the original code save for a few small known exceptions. This accounts for all the quirks and optimisations in the interpreter. Currently it works in almost all cases for the following `ast` nodes: - `Call`, e.g. `self.foo(bar)` - `Attribute`, e.g. `point.x` - `Subscript`, e.g. `lst[1]` - `BinOp`, e.g. `x + y` (doesn't include `and` and `or`) - `UnaryOp`, e.g. `-n` (includes `not` but only works sometimes) - `Compare` e.g. `a < b` (not for chains such as `0 < p < 1`) The plan is to extend to more operations in the future. - **[`stack_data`](https://github.com/alexmojaki/stack_data)**: Extracts data from stack frames and tracebacks, particularly to display more useful tracebacks than the default. Also uses another related library of mine: **[`pure_eval`](https://github.com/alexmojaki/pure_eval)**. - **[`futurecoder`](https://futurecoder.io/)**: Highlights the executing node in tracebacks using `executing` via `stack_data`, and provides debugging with `snoop`. - **[`snoop`](https://github.com/alexmojaki/snoop)**: A feature-rich and convenient debugging library. Uses `executing` to show the operation which caused an exception and to allow the `pp` function to display the source of its arguments. - **[`heartrate`](https://github.com/alexmojaki/heartrate)**: A simple real time visualisation of the execution of a Python program. Uses `executing` to highlight currently executing operations, particularly in each frame of the stack trace. - **[`sorcery`](https://github.com/alexmojaki/sorcery)**: Dark magic delights in Python. Uses `executing` to let special callables called spells know where they're being called from. - **[`IPython`](https://github.com/ipython/ipython/pull/12150)**: Highlights the executing node in tracebacks using `executing` via [`stack_data`](https://github.com/alexmojaki/stack_data). - **[`icecream`](https://github.com/gruns/icecream)**: 🍦 Sweet and creamy print debugging. Uses `executing` to identify where `ic` is called and print its arguments. - **[`friendly_traceback`](https://github.com/friendly-traceback/friendly-traceback)**: Uses `stack_data` and `executing` to pinpoint the cause of errors and provide helpful explanations. - **[`python-devtools`](https://github.com/samuelcolvin/python-devtools)**: Uses `executing` for print debugging similar to `icecream`. - **[`sentry_sdk`](https://github.com/getsentry/sentry-python)**: Add the integration `sentry_sdk.integrations.executingExecutingIntegration()` to show the function `__qualname__` in each frame in sentry events. - **[`varname`](https://github.com/pwwang/python-varname)**: Dark magics about variable names in python. Uses `executing` to find where its various magical functions like `varname` and `nameof` are called from.
Module to annotate Python abstract syntax trees with source code positions
The asttokens module annotates Python abstract syntax trees (ASTs) with the positions of tokens and text in the source code that generated them. This makes it possible for tools that work with logical AST nodes to find the particular text that resulted in those nodes, for example for automated refactoring or highlighting.
Safely evaluate AST nodes without side effects
[![Build Status](https://travis-ci.org/alexmojaki/pure_eval.svg?branch=master)](https://travis-ci.org/alexmojaki/pure_eval) [![Coverage Status](https://coveralls.io/repos/github/alexmojaki/pure_eval/badge.svg?branch=master)](https://coveralls.io/github/alexmojaki/pure_eval?branch=master) [![Supports Python versions 3.5+](https://img.shields.io/pypi/pyversions/pure_eval.svg)](https://pypi.python.org/pypi/pure_eval) This is a Python package that lets you safely evaluate certain AST nodes without triggering arbitrary code that may have unwanted side effects. It can be installed from PyPI: pip install pure_eval To demonstrate usage, suppose we have an object defined as follows: ```python class Rectangle: def __init__(self, width, height): self.width = width self.height = height @property def area(self): print("Calculating area...") return self.width * self.height rect = Rectangle(3, 5) ``` Given the `rect` object, we want to evaluate whatever expressions we can in this source code: ```python source = "(rect.width, rect.height, rect.area)" ``` This library works with the AST, so let's parse the source code and peek inside: ```python import ast tree = ast.parse(source) the_tuple = tree.body[0].value for node in the_tuple.elts: print(ast.dump(node)) ``` Output: ```python Attribute(value=Name(id='rect', ctx=Load()), attr='width', ctx=Load()) Attribute(value=Name(id='rect', ctx=Load()), attr='height', ctx=Load()) Attribute(value=Name(id='rect', ctx=Load()), attr='area', ctx=Load()) ``` Now to actually use the library. First construct an Evaluator: ```python from pure_eval import Evaluator evaluator = Evaluator({"rect": rect}) ``` The argument to `Evaluator` should be a mapping from variable names to their values. Or if you have access to the stack frame where `rect` is defined, you can instead use: ```python evaluator = Evaluator.from_frame(frame) ``` Now to evaluate some nodes, using `evaluator[node]`: ```python print("rect.width:", evaluator[the_tuple.elts[0]]) print("rect:", evaluator[the_tuple.elts[0].value]) ``` Output: ``` rect.width: 3 rect: <__main__.Rectangle object at 0x105b0dd30> ``` OK, but you could have done the same thing with `eval`. The useful part is that it will refuse to evaluate the property `rect.area` because that would trigger unknown code. If we try, it'll raise a `CannotEval` exception. ```python from pure_eval import CannotEval try: print("rect.area:", evaluator[the_tuple.elts[2]]) # fails except CannotEval as e: print(e) # prints CannotEval ``` To find all the expressions that can be evaluated in a tree: ```python for node, value in evaluator.find_expressions(tree): print(ast.dump(node), value) ``` Output: ```python Attribute(value=Name(id='rect', ctx=Load()), attr='width', ctx=Load()) 3 Attribute(value=Name(id='rect', ctx=Load()), attr='height', ctx=Load()) 5 Name(id='rect', ctx=Load()) <__main__.Rectangle object at 0x105568d30> Name(id='rect', ctx=Load()) <__main__.Rectangle object at 0x105568d30> Name(id='rect', ctx=Load()) <__main__.Rectangle object at 0x105568d30> ``` Note that this includes `rect` three times, once for each appearance in the source code. Since all these nodes are equivalent, we can group them together: ```python from pure_eval import group_expressions for nodes, values in group_expressions(evaluator.find_expressions(tree)): print(len(nodes), "nodes with value:", values) ``` Output: ``` 1 nodes with value: 3 1 nodes with value: 5 3 nodes with value: <__main__.Rectangle object at 0x10d374d30> ``` If we want to list all the expressions in a tree, we may want to filter out certain expressions whose values are obvious. For example, suppose we have a function `foo`: ```python def foo(): pass ``` If we refer to `foo` by its name as usual, then that's not interesting: ```python from pure_eval import is_expression_interesting node = ast.parse('foo').body[0].value print(ast.dump(node)) print(is_expression_interesting(node, foo)) ``` Output: ```python Name(id='foo', ctx=Load()) False ``` But if we refer to it by a different name, then it's interesting: ```python node = ast.parse('bar').body[0].value print(ast.dump(node)) print(is_expression_interesting(node, foo)) ``` Output: ```python Name(id='bar', ctx=Load()) True ``` In general `is_expression_interesting` returns False for the following values: - Literals (e.g. `123`, `'abc'`, `[1, 2, 3]`, `{'a': (), 'b': ([1, 2], [3])}`) - Variables or attributes whose name is equal to the value's `__name__`, such as `foo` above or `self.foo` if it was a method. - Builtins (e.g. `len`) referred to by their usual name. To make things easier, you can combine finding expressions, grouping them, and filtering out the obvious ones with: ```python evaluator.interesting_expressions_grouped(root) ``` To get the source code of an AST node, I recommend [asttokens](https://github.com/gristlabs/asttokens). Here's a complete example that brings it all together: ```python from asttokens import ASTTokens from pure_eval import Evaluator source = """ x = 1 d = {x: 2} y = d[x] """ names = {} exec(source, names) atok = ASTTokens(source, parse=True) for nodes, value in Evaluator(names).interesting_expressions_grouped(atok.tree): print(atok.get_text(nodes[0]), "=", value) ``` Output: ```python x = 1 d = {1: 2} y = 2 d[x] = 2 ```
Interpreter of the Python3 programming language
Python combines remarkable power with very clear syntax. It has modules, classes, exceptions, very high level dynamic data types, and dynamic typing. There are interfaces to many system calls and libraries, as well as to various windowing systems. New built-in modules are easily written in C or C++ (or other languages, depending on the chosen implementation). Python is also usable as an extension language for applications written in other languages that need easy-to-use scripting or automation interfaces. This package Provides python version 3.