Merge 4-0-121 to backup_sync #127
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
They may fail throwing an unrecoverable exception in case of unresolved includes etc. Also, some minor clean-up.
This PR handles: * Move parser tests from the `deprecated` frontend to the new `ruby` frontend. * Fixed parser issues with HashLiterals * Fixed parser issues with one-liner class definitions * Fixed parser issues with arguments in functions
Brings in latest babel and typescript to astgen.
* [ruby] Arrow Lambda Parameter Fix Tests various lambdas and fixes `self` parameter name in methods.
partly to minify the flatgraph diff
* minor work on reachingDef * fmt --------- Co-authored-by: Michael Pollmeier <[email protected]>
This astgen version skipps giant, unparsable files with EMSCRIPTEN code now by default. For: https://shiftleftinc.atlassian.net/browse/SEN-2797
Also handle fullnames with generics correctly when stubbing types For: https://shiftleftinc.atlassian.net/browse/SEN-2840
This PR adds a few more parser tests based on tests in the `querying/` folder for Ruby.
Inheritance via `<` in Ruby can be arbitrary extensions which warrant post-processing analysis, so this removes any attempt to resolve the type at AST creation to allow for a post-processing pass to handle this instead.
Does not exist anymore.
As the `<body>` call is synthetic and meant to be immediately deterministic, so there is no reason it should be re-determined.
* upgrade cpg and adapt * use released cpg
…joernio#4729) ``` [warn] -- [E092] Pattern Match Unchecked Warning: /home/mp/Projects/shiftleft/joern.1/joern-cli/frontends/pysrc2cpg/src/test/scala/io/joern/pysrc2cpg/PySrc2CpgFixture.scala:74:61 [warn] 74 | path.resultPairs().collect { case (firstElement: String, secondElement: Option[Integer]) => [warn] | ^ [warn] |the type test for Option[Integer] cannot be checked at runtime because its type arguments can't be determined from Option[Int] [warn] | [warn] | longer explanation available when compiling with `-explain` } ```
Earlier, if we pass the directory path which contains multiple go modules. Processing was done with all the `.go` files mapped to single `go.mod` file. With this change, we have segregated the processing by first isolating all the files mapped to respective `go.mod`. This will also make sure to cleanup the memory footprint after every module is being processed. However, this will increase the processing when used with download dependency as it will process all the `go.mod` files for identifying and processing used dependencies.
This PR changes the modelling of Singleton methods on objects. Ex:
```ruby
class Animal; end
animal = Animal.new
class << animal
def bark
"woof"
end
end
```
Creates a lambda method for `animal.bark`, and assigns `animal.bark = methodRef(bark)`
Resolves joernio#4721
- we had one stackoverflow in fullname - evaluation.getOverload may return null
…oernio#4733) * Fix always-crashing array initializer type resolution * Fix exception when trying to cast ArrayType to ClassOrInterfaceType * Fix formatting
…g merge (joernio#4737) * minify the flatgraph diff by bringing various things in before the big merge * fmt * revert accidental change
For new frontends with potentially malformed ASTs, type recovery must safely handle instances where members don't have AST parents.
The parser emits calls to `super` as different from simple calls, this PR handles them.
Also: no more empty method fullnames
Sometimes AST trees would be thrown away, specifically when long chained expressions would be compressed into temporary cache variables. This only happens with expressions, and closures typically write directly to the diff graph. This PR minimizes what is written directly to the diff graph from function creation, and extends `x2cpg.Ast` to also include `CAPTURE` edges. This also introduces `closureToRefs` map to track which `Block` nodes already have live method ASTs in the diff graph.
…rnio#4983) This reverts commit 3c27bf6.
* [ruby] Changed handling of ForEach loops * [ruby] fixed double _astIn reference to identifier * [ruby] fix failing tests
…" (joernio#4985) * Revert "Revert "[ruby] Ignore "Throwaway" AST Structures (joernio#4982)" (joernio#4983)" This reverts commit 464480d. * [ruby] Re-implemented "Ignore "Throwaway" AST Structures (joernio#4982)" This correctly prevents re-use of nodes that are already being used elsewhere by ensuring deep copies.
`RubyExpression` nodes don't inherently consider the `span` in the calculation of its hash, so when put into any hashed context, nodes that only rely on `span` alone will collide in these contexts.
… dir (joernio#4956) The logic to guess the base dir of the installation is quite fiddly but works for our use cases for astgen. PhpParser implemented something similar, but not quite - and it failed for buildbot. On buildbot the installation path for php2cpg is `/worker/sptestV2-php2cpg/build/x2cpg-internal/php2cpg/target/universal/stage` which (prior to this PR) leads to an invalid derived php-parser name and the following error: ``` 2024-09-25 09:30:08.623 ERROR Invalid path for PhpParserBin: /worker/sptestV2-/php2cpg/bin/php-parser/php-parser.php ```
…nse header (joernio#4995) Turns out the HTTPServer expects an explicit "Connection: close" header which requests the connection to be closed after the transaction ends. Otherwise, it would wait for a 10sec timeout for the next thread to become available. In case we only allow for one thread (jimple2cpg) that would mean additional waiting which renders the whole server approach useless. This in now fixes as we immediately close the connection after the frontend is finished.
This happened for parameter and variable declarations in parentheses.
…ex for named arguments (joernio#5000)
* upgrade deps * trigger ci
…ed arguments (joernio#5003) * [dataflowengineoss] Fix PassThroughMapping criteria for named arguments * sort result for testing purposes
joernio#4998) DescriptorRendererImpl asserts that the options are locked upon init. This means that the type rendering will fail anywhere with assertions enabled (`-ea`) as the options weren't being locked after setup.
* ExternalCommand: more information in error conditions * report exit code if it's non-zero * pass on original error (if any) rather than disregarding it * log.warn stderr output (if any) * add tests * compiler warning fix * exit code `2` on linux, `1` on mac... * fix for mac * error msg is different on windows
Works with Java Process / ProcessBuilder now. No more scala.sys.process. ----------- Co-authored-by: Michael Pollmeier <[email protected]>
fixes the regression from the ExternalCommand refactor, but I couldn't leave the the env var handling as it was either. Fixing that by using a proper library for parsing CLI arguments
* [kotlin2cpg] Improve fullnames. Improve and cleanup the fullname calculation for methods and callsites. The most noteable changes here are: 1. The lambda function naming schema changed from `package.<fileName>.<lambda>counter:signature` to `package.scope.<lambda>counter:signature`. This give a better indication which lambda function is represented also gets rid of exception of using the filename in the method fullnames. 2. Fixed lambda function signatures. 3. The fullname schema of extension methods changed from `extendedClassFullName.extensionMethodName:signatureWithoutThisParameter` to `package.extensionMethodName:signatureWithThisParameter` The fact that the this parameter is part of the signature is an exception in our java fullnames but necessary in order to distinguish otherwise identical extension methods. 4. We avoid the recalculation of the binding context with every file processed. This is now done once at the begining. 5. Calls to `::class` are now represented as calls to `<operator>.class`. * Fix extension method arg and param indicies. The instance arg/param now have index 1 as they are not the receiver on which the call is dispatches. It is a static call after all. * Implement first sane version of lambda binding nodes. Additionally fixed the inheritance information for lambda type decl nodes. * Refactor NameRender. Split some methods into BindingContextUtils and renamed those methods. * fmt * Incorporate review comments. - No more special case for handling `kotlin.Function`. - Removed some unused code. - Stop using currentTimeMillis. * Remove typeInfoProvider.expressionType and replace it via nameRenderer. This results in expresssion types now being renderer in the new fassion. * Translate kotlin.Array type full name to java representation. * Removed multiple further type full name API methods from TypeInfoProvider. * Remove typeFullName for KtDestructuringDeclarationEntry from TypeInfoProvider. * Remove visibility and modality API from TypeInfoProvider. * Remove containingTypeDeclFullName. * Remove unnecessary code. * Remove inheritanceTypes API from TypeInfoProvider. * Remove destructuringEntries API from TypeInfoProvider. * Removed KtTypeAlias related TypeInfoProvider APIs. * Remove typeFullName API for binary expressions from TypeInfoProvider. * Remove typeFullName for annotations from TypeInfoProvider. * Remove unnecessary Option types from BindingContextUtils APIs. * Rework lambda implicit parameter handling. - Removed implicitParameterName and hasApplyOrAlsoScopeFunctionParent APIs from TypeInfoProvider. - Removed wrong parameter deconstruction in lambda. New implemention is missing. - Some lambda to builtin `apply` and `also` functions did not get return statements generated. That is now fixed. * Remove typeFullName and isCompanionObject APIs from TypeInfoProvider. * Remove typeFullName API for KtTypeReference from TypeInfoProvider. * Remove containingDeclType API from TypeInfoProvider. * Remove typeFullName and referenceTargetTypeFullName APIs from TypeInfoProvider. Also refactored assignmentAstForDestructuringEntry. The right hand side base is now created outside of this function allowing for other constructs than pure identifiers. The pure identifier were e.g. wrong in case of class member references. * Remove propertyType and typeFromImports API from TypeInfoProvider. * Remove TypeRenderer. * Improve comment. * Fmt. * Fix for `ScopeFunctionsTests` * Adjusted test expectations for more precise/sound outcomes * Left note about the lambda flows * Add better fallback handling methodFullName of calls. In case of incomplete type information calls with overloads can get ambiguous. We now use the ambiguous function descriptors to get as much of a methodFullName correct as possible. * Fix test fullname. --------- Co-authored-by: David Baker Effendi <[email protected]>
This bypasses tty buffering and read timeouts.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Created by Github action