Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Comprehensions of Lists, Sets and Maps and generator expressions #1786

Merged
merged 42 commits into from
Oct 30, 2024
Merged

Add Comprehensions of Lists, Sets and Maps and generator expressions #1786

merged 42 commits into from
Oct 30, 2024

Conversation

KuechA
Copy link
Contributor

@KuechA KuechA commented Oct 10, 2024 •
edited by konradweiss
Loading

This PR aims to enable list comprehensions and similar constructs.

Tasks:

  • Add a node which will hold the comprehension
  • Add translation for Python.AST.ListComp
  • Add translation for Python.AST.SetComp
  • Add translation for Python.AST.DictComp
  • Add translation for Python.AST.GeneratorExp
  • Connect the EOG of the new nodes
  • Connect the DFG of the new nodes in DFGPass
  • Handle the new nodes in the ControlFlowSensitiveDFGPass
  • Add some tests

Fixes #1714
Fixes #1715
Fixes #1716
Fixes #1717

@codecov Codecov
Copy link

codecov bot commented Oct 10, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 75.91837% with 59 lines in your changes missing coverage. Please review.

Project coverage is 75.23%. Comparing base (d2d7af9) to head (fb30be7).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
.../statements/expressions/CollectionComprehension.kt 50.00% 17 Missing and 2 partials ⚠️
.../statements/expressions/ComprehensionExpression.kt 59.09% 16 Missing and 2 partials ⚠️
...er/aisec/cpg/frontends/python/ExpressionHandler.kt 87.03% 5 Missing and 2 partials ⚠️
...in/de/fraunhofer/aisec/cpg/graph/builder/Fluent.kt 66.66% 2 Missing and 4 partials ⚠️
...er/aisec/cpg/passes/ControlFlowSensitiveDFGPass.kt 81.81% 3 Missing and 1 partial ⚠️
...de/fraunhofer/aisec/cpg/graph/ExpressionBuilder.kt 83.33% 2 Missing ⚠️
...hofer/aisec/cpg/passes/EvaluationOrderGraphPass.kt 95.12% 2 Missing ⚠️
...n/kotlin/de/fraunhofer/aisec/cpg/passes/DFGPass.kt 92.30% 0 Missing and 1 partial ⚠️
Additional details and impacted files
Files with missing lines Coverage Δ
...de/fraunhofer/aisec/cpg/graph/edges/ast/AstEdge.kt 100.00% <ø> (ø)
...c/cpg/graph/edges/collections/UnwrappedEdgeList.kt 50.00% <100.00%> (+4.34%) ⬆️
...n/kotlin/de/fraunhofer/aisec/cpg/passes/DFGPass.kt 83.24% <92.30%> (+0.71%) ⬆️
...de/fraunhofer/aisec/cpg/graph/ExpressionBuilder.kt 81.91% <83.33%> (+0.06%) ⬆️
...hofer/aisec/cpg/passes/EvaluationOrderGraphPass.kt 89.62% <95.12%> (+0.69%) ⬆️
...er/aisec/cpg/passes/ControlFlowSensitiveDFGPass.kt 78.51% <81.81%> (+0.29%) ⬆️
...in/de/fraunhofer/aisec/cpg/graph/builder/Fluent.kt 86.48% <66.66%> (-0.74%) ⬇️
...er/aisec/cpg/frontends/python/ExpressionHandler.kt 75.09% <87.03%> (+3.97%) ⬆️
.../statements/expressions/ComprehensionExpression.kt 59.09% <59.09%> (ø)
.../statements/expressions/CollectionComprehension.kt 50.00% <50.00%> (ø)

... and 4 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@KuechA
Copy link
Contributor Author

KuechA commented Oct 18, 2024

@konradweiss please check and update the EOG implementation and specs

@KuechA KuechA changed the title Add Comprehensions of Lists, Sets and Maps Add Comprehensions of Lists, Sets and Maps and generator expressions Oct 18, 2024
@KuechA KuechA added the python label Oct 18, 2024
@KuechA KuechA marked this pull request as ready for review October 24, 2024 12:46
konradweiss
konradweiss approved these changes Oct 24, 2024
View reviewed changes
Copy link
Collaborator

@konradweiss konradweiss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

maximiliankaul
maximiliankaul approved these changes Oct 29, 2024
View reviewed changes
cpg-core/src/main/kotlin/de/fraunhofer/aisec/cpg/graph/builder/Fluent.kt Show resolved Hide resolved
cpg-core/src/main/kotlin/de/fraunhofer/aisec/cpg/graph/builder/Fluent.kt Show resolved Hide resolved
cpg-core/src/main/kotlin/de/fraunhofer/aisec/cpg/graph/edges/collections/UnwrappedEdgeList.kt Show resolved Hide resolved
cpg-core/src/test/kotlin/de/fraunhofer/aisec/cpg/graph/FluentTest.kt Outdated Show resolved Hide resolved
cpg-core/src/test/kotlin/de/fraunhofer/aisec/cpg/graph/FluentTest.kt Outdated Show resolved Hide resolved
...age-python/src/test/kotlin/de/fraunhofer/aisec/cpg/frontends/python/ExpressionHandlerTest.kt Outdated Show resolved Hide resolved
.../main/kotlin/de/fraunhofer/aisec/cpg/graph/statements/expressions/ComprehensionExpression.kt Outdated Show resolved Hide resolved
.../main/kotlin/de/fraunhofer/aisec/cpg/graph/statements/expressions/ComprehensionExpression.kt Show resolved Hide resolved
.../main/kotlin/de/fraunhofer/aisec/cpg/graph/statements/expressions/ComprehensionExpression.kt Show resolved Hide resolved
.../main/kotlin/de/fraunhofer/aisec/cpg/graph/statements/expressions/ComprehensionExpression.kt Show resolved Hide resolved
@maximiliankaul maximiliankaul mentioned this pull request Oct 29, 2024
Merged
@KuechA KuechA merged commit 884c577 into main Oct 30, 2024
5 checks passed
@KuechA KuechA deleted the ak/slice-all-collections branch October 30, 2024 17:27
lshala pushed a commit that referenced this pull request Nov 8, 2024
@KuechA @konradweiss @lshala
Add Comprehensions of Lists, Sets and Maps and generator expressions (#…
84e331f 
…1786)

* Add comprehension expression

* Initial python translation for listcomp

* SetComp and DictComp in python frontend

* First simple tests

* test in main

* Try to add DFG edges

* Fix not implemented error

* Fix more bugs

* Also handle GeneratorExp, add some documentation.

* Extract nested class to own file

* Fix bug, aggregate predicates

* Remove unnecessary changes

* Specify idea for EOG

* Fake higher test coverage

* More testing

* More tests

* Fix error from renaming

* Handle the comprehension expression in the control flow sensitive DFG

* Adding alternatives to EOG for collection comprehension and fixing syntax error in comprehension expression

* Adding alternative that properly depicts generator behavior

* Small fix

* Alternative for ComprehensionExpression

* Fix

* Adding EOG handling for ComprehensionExpression and CollectionComprehension

* Add test and fix EOG pass implementation

* Allow th addition to something that holds arguments and something that holds statements

* Remove useless stuff from ControlflowSensitiveDFGPass

* Make non-optional things non-optional

* Fix test

* Remove condition to reduce code which needs coverage

* More tests

* Update stuff

* review

* review

* generator type

---------

Co-authored-by: Konrad Weiss <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@maximiliankaul maximiliankaul maximiliankaul approved these changes

@konradweiss konradweiss konradweiss approved these changes

@lshala lshala Awaiting requested review from lshala lshala is a code owner

@oxisto oxisto Awaiting requested review from oxisto oxisto is a code owner

Assignees
No one assigned
Labels
python
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

DictComp SetComp ListComp GeneratorExp
3 participants
@KuechA @konradweiss @maximiliankaul