Merge 4-0-110 to backup_sync #114
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
They may fail throwing an unrecoverable exception in case of unresolved includes etc. Also, some minor clean-up.
This PR handles: * Move parser tests from the `deprecated` frontend to the new `ruby` frontend. * Fixed parser issues with HashLiterals * Fixed parser issues with one-liner class definitions * Fixed parser issues with arguments in functions
Brings in latest babel and typescript to astgen.
* [ruby] Arrow Lambda Parameter Fix Tests various lambdas and fixes `self` parameter name in methods.
partly to minify the flatgraph diff
* minor work on reachingDef * fmt --------- Co-authored-by: Michael Pollmeier <[email protected]>
This astgen version skipps giant, unparsable files with EMSCRIPTEN code now by default. For: https://shiftleftinc.atlassian.net/browse/SEN-2797
Also handle fullnames with generics correctly when stubbing types For: https://shiftleftinc.atlassian.net/browse/SEN-2840
This PR adds a few more parser tests based on tests in the `querying/` folder for Ruby.
Inheritance via `<` in Ruby can be arbitrary extensions which warrant post-processing analysis, so this removes any attempt to resolve the type at AST creation to allow for a post-processing pass to handle this instead.
Does not exist anymore.
As the `<body>` call is synthetic and meant to be immediately deterministic, so there is no reason it should be re-determined.
* upgrade cpg and adapt * use released cpg
…joernio#4729) ``` [warn] -- [E092] Pattern Match Unchecked Warning: /home/mp/Projects/shiftleft/joern.1/joern-cli/frontends/pysrc2cpg/src/test/scala/io/joern/pysrc2cpg/PySrc2CpgFixture.scala:74:61 [warn] 74 | path.resultPairs().collect { case (firstElement: String, secondElement: Option[Integer]) => [warn] | ^ [warn] |the type test for Option[Integer] cannot be checked at runtime because its type arguments can't be determined from Option[Int] [warn] | [warn] | longer explanation available when compiling with `-explain` } ```
Earlier, if we pass the directory path which contains multiple go modules. Processing was done with all the `.go` files mapped to single `go.mod` file. With this change, we have segregated the processing by first isolating all the files mapped to respective `go.mod`. This will also make sure to cleanup the memory footprint after every module is being processed. However, this will increase the processing when used with download dependency as it will process all the `go.mod` files for identifying and processing used dependencies.
This PR changes the modelling of Singleton methods on objects. Ex:
```ruby
class Animal; end
animal = Animal.new
class << animal
def bark
"woof"
end
end
```
Creates a lambda method for `animal.bark`, and assigns `animal.bark = methodRef(bark)`
Resolves joernio#4721
- we had one stackoverflow in fullname - evaluation.getOverload may return null
…oernio#4733) * Fix always-crashing array initializer type resolution * Fix exception when trying to cast ArrayType to ClassOrInterfaceType * Fix formatting
…g merge (joernio#4737) * minify the flatgraph diff by bringing various things in before the big merge * fmt * revert accidental change
For new frontends with potentially malformed ASTs, type recovery must safely handle instances where members don't have AST parents.
The parser emits calls to `super` as different from simple calls, this PR handles them.
Also: no more empty method fullnames
There were some unused parser rules adding unnecessary complexity to the parser
Added `simpleCommandArgumentList` to allow multiple call args for `methodInvocationWithoutParentheses`
* fix assignmentModulo
* fix assignmentXor
* fix assignmentOr
* fix assignmentAnd
* fix assignmentExponentiation
* fix assignmentShiftLeft
* fix assignment{Logical,Arithmetic}ShiftRight
* scalafmt
…#4954) These synthetic METHOD nodes did not have an offset at all. For: https://shiftleftinc.atlassian.net/browse/SEN-3250
…o#4955) As soon as `FullNameSemantics` started logging duplicate semantics (in joernio#4954), it was observed that `SemanticTestCpg` was passing a fresh Semantics instance to `EngineContext` instead of passing the one used by `OssDataFlow`. By coincidence, however, they were the same. Nevertheless, this patch guarantees they are the same.
Using the CDT API (getContainingFilename) now. For: joernio#4924
* [ruby] ANTLR Profiler Summary * Introduced global profiling to summarize rule and parse performance across the project * Added a shutdown hook to dump a summary of the profiler rules in a file `antlr_summary.log` at the root of the project with this information * Check parent exists before dumping. May not be there during test cases * Added note around shutdown hook
Sometimes AST trees would be thrown away, specifically when long chained expressions would be compressed into temporary cache variables. This only happens with expressions, and closures typically write directly to the diff graph. This PR minimizes what is written directly to the diff graph from function creation, and extends `x2cpg.Ast` to also include `CAPTURE` edges. This also introduces `closureToRefs` map to track which `Block` nodes already have live method ASTs in the diff graph.
…rnio#4983) This reverts commit 3c27bf6.
* [ruby] Changed handling of ForEach loops * [ruby] fixed double _astIn reference to identifier * [ruby] fix failing tests
…" (joernio#4985) * Revert "Revert "[ruby] Ignore "Throwaway" AST Structures (joernio#4982)" (joernio#4983)" This reverts commit 464480d. * [ruby] Re-implemented "Ignore "Throwaway" AST Structures (joernio#4982)" This correctly prevents re-use of nodes that are already being used elsewhere by ensuring deep copies.
`RubyExpression` nodes don't inherently consider the `span` in the calculation of its hash, so when put into any hashed context, nodes that only rely on `span` alone will collide in these contexts.
… dir (joernio#4956) The logic to guess the base dir of the installation is quite fiddly but works for our use cases for astgen. PhpParser implemented something similar, but not quite - and it failed for buildbot. On buildbot the installation path for php2cpg is `/worker/sptestV2-php2cpg/build/x2cpg-internal/php2cpg/target/universal/stage` which (prior to this PR) leads to an invalid derived php-parser name and the following error: ``` 2024-09-25 09:30:08.623 ERROR Invalid path for PhpParserBin: /worker/sptestV2-/php2cpg/bin/php-parser/php-parser.php ```
…nse header (joernio#4995) Turns out the HTTPServer expects an explicit "Connection: close" header which requests the connection to be closed after the transaction ends. Otherwise, it would wait for a 10sec timeout for the next thread to become available. In case we only allow for one thread (jimple2cpg) that would mean additional waiting which renders the whole server approach useless. This in now fixes as we immediately close the connection after the frontend is finished.
This happened for parameter and variable declarations in parentheses.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Created by Github action