Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Xray Source Control Service #940

Merged
merged 106 commits into from
Sep 11, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
106 commits
Select commit Hold shift + click to select a range
5c110c7
add new attributes to technologiesData
EyalDelarea Feb 23, 2023
9dc0c98
add installation command to poetry package manager
EyalDelarea Feb 23, 2023
cc80c8a
add packageVersionOperator command to poetry package manager
EyalDelarea Feb 23, 2023
4c3dd20
Merge branch 'master' of https://github.com/jfrog/jfrog-cli-core
EyalDelarea Mar 5, 2023
813c79a
Merge branch 'dev' of https://github.com/jfrog/jfrog-cli-core
EyalDelarea Mar 7, 2023
6ad8772
Merge branch 'master' of https://github.com/jfrog/jfrog-cli-core
EyalDelarea Mar 18, 2023
4d20901
Upgrade jfrog-cli-core to 2.31.1 (#739)
omerzi Apr 5, 2023
9b4006e
Added the Frogbot badge to the README (#745)
eyalbe4 Apr 13, 2023
1381a8b
Merge remote-tracking branch 'upstream/dev' into mastercore
sverdlov93 Apr 19, 2023
f8e3a22
Merge remote-tracking branch 'upstream/dev' into mastercore
sverdlov93 Apr 19, 2023
4f7259c
Merge remote-tracking branch 'origin/dev'
omerzi May 2, 2023
932793d
Merge remote-tracking branch 'origin/dev'
omerzi May 3, 2023
2ddeac0
Merge branch 'dev'
eyalbe4 May 16, 2023
11ce1e8
Merge branch 'dev' into master
talarian1 May 17, 2023
521d832
Merge branch 'dev'
talarian1 May 17, 2023
f470da2
Merge remote-tracking branch 'origin/dev'
talarian1 May 17, 2023
2c85766
Merge remote-tracking branch 'origin/dev'
omerzi May 21, 2023
ae0b91d
Merge remote-tracking branch 'origin/dev'
talarian1 May 23, 2023
5a53932
Merge remote-tracking branch 'upstream/dev'
yahavi May 23, 2023
c07d9fb
Merge remote-tracking branch 'upstream/dev'
yahavi May 23, 2023
7040b6a
Merge remote-tracking branch 'origin/dev'
talarian1 May 24, 2023
e91d430
Merge branch 'dev'
eyalbe4 May 26, 2023
2680fdb
Upgrade jfrog-cli-core to 2.34.6
talarian1 May 28, 2023
b4ddac8
Merge branch 'master' into dev
eyalbe4 May 31, 2023
6dca5e7
Merge branch 'dev'
eyalbe4 May 31, 2023
9fa7b09
Merge remote-tracking branch 'origin/dev'
talarian1 May 31, 2023
a46fb01
Merge branch 'dev'
yahavi Jun 1, 2023
94469f4
Merge branch 'dev'
eyalbe4 Jun 4, 2023
b330d82
Merge branch 'dev'
eyalbe4 Jun 10, 2023
7141b4c
Merge branch 'master' into dev
omerzi Jun 27, 2023
ec17933
Merge branch 'dev'
eyalbe4 Jun 27, 2023
83ced5a
Merge branch 'dev'
RobiNino Jun 29, 2023
9e57a49
Merge branch 'dev'
yahavi Jul 4, 2023
fe4c82a
Merge branch 'master' into dev
omerzi Jul 12, 2023
47a0f3f
Upgrade jfrog-cli-core to 2.38.0
omerzi Jul 12, 2023
40e7d2d
Remove Secrets Scanner Sacanners Names config (#862)
talarian1 Jul 16, 2023
0be3aae
Upgrade jfrog-cli-core to 2.39.0
omerzi Jul 18, 2023
0a4b112
Merge remote-tracking branch 'origin/dev'
omerzi Jul 19, 2023
64ee611
Merge remote-tracking branch 'origin/dev'
omerzi Jul 19, 2023
d32579e
Merge remote-tracking branch 'origin/dev'
omerzi Jul 23, 2023
f79600c
Merge branch 'dev'
eyalbe4 Jul 31, 2023
db385fe
Merge branch 'dev'
eyalbe4 Aug 3, 2023
8e0ea69
Merge branch 'master' of https://github.com/jfrog/jfrog-cli-core
EyalDelarea Aug 15, 2023
203a303
Refactor to interface
EyalDelarea Aug 15, 2023
0a4c6dd
Split scan service
EyalDelarea Aug 15, 2023
a3860cf
Pull dev
EyalDelarea Aug 16, 2023
c35658f
Working XSC call
EyalDelarea Aug 16, 2023
c9a6c6d
Check XSC
EyalDelarea Aug 16, 2023
26665de
Check XSC
EyalDelarea Aug 16, 2023
33c5f6a
fix multi-scan-id
EyalDelarea Aug 16, 2023
ad71ffa
Fix code analysis errors
EyalDelarea Aug 16, 2023
177bf7b
remove testings object
EyalDelarea Aug 16, 2023
76eae4b
fix always using xsc
EyalDelarea Aug 17, 2023
fd6d7d2
fix static check
EyalDelarea Aug 17, 2023
c845f56
update go.mod
EyalDelarea Aug 17, 2023
f537f57
Fix choose XSC server when enabled
EyalDelarea Aug 17, 2023
3f8ba64
Handle XSC/XRAY url provided from flags and config.
EyalDelarea Aug 17, 2023
2703ea9
update go.mod
EyalDelarea Aug 17, 2023
15b1b89
Refactor
EyalDelarea Aug 17, 2023
7f03d26
Set tech after detection
EyalDelarea Aug 20, 2023
54e9636
pull dev
EyalDelarea Aug 20, 2023
7f29321
Update deps
EyalDelarea Aug 21, 2023
da848fe
update deps
EyalDelarea Aug 21, 2023
1c5b610
pull dev
EyalDelarea Aug 22, 2023
c06269d
move xsc endpoint const
EyalDelarea Aug 22, 2023
6e5b7f7
pull dev
EyalDelarea Aug 23, 2023
764490c
Update deps
EyalDelarea Aug 23, 2023
40a7068
Merge remote-tracking branch 'upstream/dev' into xsc
EyalDelarea Aug 24, 2023
8aa7d33
merge
EyalDelarea Aug 24, 2023
2dcae54
pass multi scan id as extra arg
EyalDelarea Aug 27, 2023
74017fb
Merge remote-tracking branch 'upstream/dev' into xsc
EyalDelarea Aug 27, 2023
35af079
pull dev
EyalDelarea Aug 27, 2023
a2e0bec
fix new arg
EyalDelarea Aug 27, 2023
f524051
pull dev
EyalDelarea Aug 28, 2023
e1ab392
pull dev
EyalDelarea Aug 29, 2023
64b367d
update deps
EyalDelarea Aug 29, 2023
196ae26
Remove testings code, add flag to provide multi scan id
EyalDelarea Aug 30, 2023
15d7d74
pull dev
EyalDelarea Aug 31, 2023
d476255
pass multi-scan-id
EyalDelarea Aug 31, 2023
7a49915
remove print
EyalDelarea Aug 31, 2023
5dcc2fa
update deps and log for debugging
EyalDelarea Aug 31, 2023
df51758
merge dev
EyalDelarea Sep 3, 2023
ccb508c
Add debug logs
EyalDelarea Sep 3, 2023
688e693
pull dev
EyalDelarea Sep 4, 2023
623cdbd
fix MSI flag
EyalDelarea Sep 4, 2023
2f79c0d
pull dev
EyalDelarea Sep 5, 2023
570d81c
fix pull dev
EyalDelarea Sep 5, 2023
5f3ac87
add xsc min version
EyalDelarea Sep 5, 2023
a9521b4
pull dev
EyalDelarea Sep 6, 2023
8455ea2
Merge branch 'dev' of https://github.com/jfrog/jfrog-cli-core into xsc
EyalDelarea Sep 6, 2023
80f33a6
update jfrog-client-go
EyalDelarea Sep 6, 2023
6ed3b67
refactor
EyalDelarea Sep 7, 2023
9c23e7a
Fix comments and remove TODO
EyalDelarea Sep 7, 2023
8215745
pull client
EyalDelarea Sep 8, 2023
74ff464
pull dev
EyalDelarea Sep 8, 2023
1a8083e
update deps
EyalDelarea Sep 8, 2023
fd3454b
CR
EyalDelarea Sep 10, 2023
e3e3e88
update deps
EyalDelarea Sep 10, 2023
ce13fce
Merge branch 'dev' of https://github.com/jfrog/jfrog-cli-core into xsc
EyalDelarea Sep 10, 2023
d91f3ee
add xsc
EyalDelarea Sep 10, 2023
f1d0181
fix static check
EyalDelarea Sep 10, 2023
a8356f8
CR
EyalDelarea Sep 10, 2023
8a7ae1b
CR
EyalDelarea Sep 11, 2023
0a29af6
pull dev
EyalDelarea Sep 11, 2023
d25a431
go fmt & update deps
EyalDelarea Sep 11, 2023
f149a70
update deps add extra check
EyalDelarea Sep 11, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ require (
github.com/jedib0t/go-pretty/v6 v6.4.7
github.com/jfrog/build-info-go v1.9.10
github.com/jfrog/gofrog v1.3.0
github.com/jfrog/jfrog-client-go v1.31.6
github.com/jfrog/jfrog-client-go v1.32.1
github.com/magiconair/properties v1.8.7
github.com/manifoldco/promptui v0.9.0
github.com/owenrumney/go-sarif/v2 v2.2.0
Expand Down Expand Up @@ -93,8 +93,6 @@ require (
gopkg.in/warnings.v0 v0.1.2 // indirect
)

replace github.com/jfrog/jfrog-client-go => github.com/jfrog/jfrog-client-go v1.28.1-0.20230910192358-6994626b2069

// replace github.com/jfrog/build-info-go => github.com/jfrog/build-info-go v1.8.9-0.20230905120411-62d1bdd4eb38

// replace github.com/jfrog/gofrog => github.com/jfrog/gofrog v1.2.6-0.20230418122323-2bf299dd6d27
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -198,8 +198,8 @@ github.com/jfrog/build-info-go v1.9.10 h1:uXnDLVxpqxoAMpXcki00QaBB+M2BoGMMpHODPk
github.com/jfrog/build-info-go v1.9.10/go.mod h1:ujJ8XQZMdT2tMkLSMJNyDd1pCY+duwHdjV+9or9FLIg=
github.com/jfrog/gofrog v1.3.0 h1:o4zgsBZE4QyDbz2M7D4K6fXPTBJht+8lE87mS9bw7Gk=
github.com/jfrog/gofrog v1.3.0/go.mod h1:IFMc+V/yf7rA5WZ74CSbXe+Lgf0iApEQLxRZVzKRUR0=
github.com/jfrog/jfrog-client-go v1.28.1-0.20230910192358-6994626b2069 h1:vk+P6jK4Zv8+F44ZnRxXUPT14BQxjJtNKdpGdemci7A=
github.com/jfrog/jfrog-client-go v1.28.1-0.20230910192358-6994626b2069/go.mod h1:362+oa7uTTYurzBs1L0dmUTlLo7uhpAU/pwM5Zb9clg=
github.com/jfrog/jfrog-client-go v1.32.1 h1:RQmuPSLsF5222vZJzwkgHSZMMJF83ExS7SwIvh4P+H8=
github.com/jfrog/jfrog-client-go v1.32.1/go.mod h1:362+oa7uTTYurzBs1L0dmUTlLo7uhpAU/pwM5Zb9clg=
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
github.com/kevinburke/ssh_config v1.2.0 h1:x584FjTGwHzMwvHx18PXxbBVzfnxogHaAReU4gf13a4=
Expand Down
5 changes: 2 additions & 3 deletions xray/commands/audit/audit.go
Original file line number Diff line number Diff line change
Expand Up @@ -158,8 +158,7 @@ func RunAudit(auditParams *AuditParams) (results *Results, err error) {
return
}
var xrayManager *xray.XrayServicesManager
xrayManager, auditParams.xrayVersion, err = xrayutils.CreateXrayServiceManagerAndGetVersion(serverDetails)
if err != nil {
if xrayManager, auditParams.xrayVersion, err = xrayutils.CreateXrayServiceManagerAndGetVersion(serverDetails); err != nil {
return
}
if err = clientutils.ValidateMinimumVersion(clientutils.Xray, auditParams.xrayVersion, scangraph.GraphScanMinXrayVersion); err != nil {
Expand Down Expand Up @@ -187,7 +186,7 @@ func RunAudit(auditParams *AuditParams) (results *Results, err error) {

// Run scanners only if the user is entitled for Advanced Security
if results.ExtendedScanResults.EntitledForJas {
results.JasError = runJasScannersAndSetResults(results.ExtendedScanResults, auditParams.DirectDependencies(), serverDetails, auditParams.workingDirs, auditParams.Progress())
results.JasError = runJasScannersAndSetResults(results.ExtendedScanResults, auditParams.DirectDependencies(), serverDetails, auditParams.workingDirs, auditParams.Progress(), auditParams.xrayGraphScanParams.MultiScanId)
}
return
}
Expand Down
5 changes: 3 additions & 2 deletions xray/commands/audit/jas/common.go
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ type JasScanner struct {
ScannerDirCleanupFunc func() error
}

func NewJasScanner(workingDirs []string, serverDetails *config.ServerDetails) (scanner *JasScanner, err error) {
func NewJasScanner(workingDirs []string, serverDetails *config.ServerDetails, multiScanId string) (scanner *JasScanner, err error) {
scanner = &JasScanner{}
if scanner.AnalyzerManager.AnalyzerManagerFullPath, err = utils.GetAnalyzerManagerExecutable(); err != nil {
return
Expand All @@ -57,6 +57,7 @@ func NewJasScanner(workingDirs []string, serverDetails *config.ServerDetails) (s
scanner.ConfigFileName = filepath.Join(tempDir, "config.yaml")
scanner.ResultsFileName = filepath.Join(tempDir, "results.sarif")
scanner.WorkingDirs, err = coreutils.GetFullPathsWorkingDirs(workingDirs)
scanner.AnalyzerManager.MultiScanId = multiScanId
return
}

Expand Down Expand Up @@ -181,7 +182,7 @@ var FakeBasicXrayResults = []services.ScanResponse{

func InitJasTest(t *testing.T, workingDirs ...string) (*JasScanner, func()) {
assert.NoError(t, rtutils.DownloadAnalyzerManagerIfNeeded())
scanner, err := NewJasScanner(workingDirs, &FakeServerDetails)
scanner, err := NewJasScanner(workingDirs, &FakeServerDetails, "")
assert.NoError(t, err)
return scanner, func() {
assert.NoError(t, scanner.ScannerDirCleanupFunc())
Expand Down
4 changes: 2 additions & 2 deletions xray/commands/audit/jasrunner.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,12 @@ import (
)

func runJasScannersAndSetResults(scanResults *utils.ExtendedScanResults, directDependencies []string,
serverDetails *config.ServerDetails, workingDirs []string, progress io.ProgressMgr) (err error) {
serverDetails *config.ServerDetails, workingDirs []string, progress io.ProgressMgr, multiScanId string) (err error) {
if serverDetails == nil || len(serverDetails.Url) == 0 {
log.Warn("To include 'Advanced Security' scan as part of the audit output, please run the 'jf c add' command before running this command.")
return
}
scanner, err := jas.NewJasScanner(workingDirs, serverDetails)
scanner, err := jas.NewJasScanner(workingDirs, serverDetails, multiScanId)
if err != nil {
return
}
Expand Down
6 changes: 3 additions & 3 deletions xray/commands/audit/jasrunner_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,22 +22,22 @@ func TestGetExtendedScanResults_AnalyzerManagerDoesntExist(t *testing.T) {
assert.NoError(t, os.Unsetenv(coreutils.HomeDir))
}()
scanResults := &utils.ExtendedScanResults{XrayResults: jas.FakeBasicXrayResults, ScannedTechnologies: []coreutils.Technology{coreutils.Yarn}}
err = runJasScannersAndSetResults(scanResults, []string{"issueId_1_direct_dependency", "issueId_2_direct_dependency"}, &jas.FakeServerDetails, nil, nil)
err = runJasScannersAndSetResults(scanResults, []string{"issueId_1_direct_dependency", "issueId_2_direct_dependency"}, &jas.FakeServerDetails, nil, nil, "")
// Expect error:
assert.Error(t, err)
}

func TestGetExtendedScanResults_ServerNotValid(t *testing.T) {
scanResults := &utils.ExtendedScanResults{XrayResults: jas.FakeBasicXrayResults, ScannedTechnologies: []coreutils.Technology{coreutils.Pip}}
err := runJasScannersAndSetResults(scanResults, []string{"issueId_1_direct_dependency", "issueId_2_direct_dependency"}, nil, nil, nil)
err := runJasScannersAndSetResults(scanResults, []string{"issueId_1_direct_dependency", "issueId_2_direct_dependency"}, nil, nil, nil, "")
assert.NoError(t, err)
}

func TestGetExtendedScanResults_AnalyzerManagerReturnsError(t *testing.T) {
mockDirectDependencies := []string{"issueId_2_direct_dependency", "issueId_1_direct_dependency"}
assert.NoError(t, rtutils.DownloadAnalyzerManagerIfNeeded())
scanResults := &utils.ExtendedScanResults{XrayResults: jas.FakeBasicXrayResults, ScannedTechnologies: []coreutils.Technology{coreutils.Yarn}}
err := runJasScannersAndSetResults(scanResults, mockDirectDependencies, &jas.FakeServerDetails, nil, nil)
err := runJasScannersAndSetResults(scanResults, mockDirectDependencies, &jas.FakeServerDetails, nil, nil, "")

// Expect error:
assert.ErrorContains(t, err, "failed to run Applicability scan")
Expand Down
4 changes: 4 additions & 0 deletions xray/commands/audit/sca/common.go
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,10 @@ func populateXrayDependencyTree(currNode *xrayUtils.GraphNode, treeHelper map[st

func RunXrayDependenciesTreeScanGraph(dependencyTree *xrayUtils.GraphNode, progress ioUtils.ProgressMgr, technology coreutils.Technology, scanGraphParams *scangraph.ScanGraphParams) (results []services.ScanResponse, err error) {
scanGraphParams.XrayGraphScanParams().DependenciesGraph = dependencyTree
xscGitInfoContext := scanGraphParams.XrayGraphScanParams().XscGitInfoContext
if xscGitInfoContext != nil {
xscGitInfoContext.Technologies = []string{technology.ToString()}
}
scanMessage := fmt.Sprintf("Scanning %d %s dependencies", len(dependencyTree.Nodes), technology)
if progress != nil {
progress.SetHeadlineMsg(scanMessage)
Expand Down
11 changes: 10 additions & 1 deletion xray/scangraph/scangraph.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,20 @@ func RunScanGraphAndGetResults(params *ScanGraphParams) (*services.ScanResponse,
// Remove scan type param if Xray version is under the minimum supported version
params.xrayGraphScanParams.ScanType = ""
}

if params.xrayGraphScanParams.XscGitInfoContext != nil {
if params.xrayGraphScanParams.XscVersion, err = xrayManager.XscEnabled(); err != nil {
return nil, err
}
}

scanId, err := xrayManager.ScanGraph(*params.xrayGraphScanParams)
if err != nil {
return nil, err
}
scanResult, err := xrayManager.GetScanGraphResults(scanId, params.XrayGraphScanParams().IncludeVulnerabilities, params.XrayGraphScanParams().IncludeLicenses)

xscEnabled := params.xrayGraphScanParams.XscVersion != ""
scanResult, err := xrayManager.GetScanGraphResults(scanId, params.XrayGraphScanParams().IncludeVulnerabilities, params.XrayGraphScanParams().IncludeLicenses, xscEnabled)
if err != nil {
return nil, err
}
Expand Down
3 changes: 2 additions & 1 deletion xray/utils/analyzermanager.go
Original file line number Diff line number Diff line change
Expand Up @@ -89,13 +89,14 @@ func (e *ExtendedScanResults) getXrayScanResults() []services.ScanResponse {

type AnalyzerManager struct {
AnalyzerManagerFullPath string
MultiScanId string
}

func (am *AnalyzerManager) Exec(configFile, scanCommand, workingDir string, serverDetails *config.ServerDetails) (err error) {
if err = SetAnalyzerManagerEnvVariables(serverDetails); err != nil {
return err
}
cmd := exec.Command(am.AnalyzerManagerFullPath, scanCommand, configFile)
cmd := exec.Command(am.AnalyzerManagerFullPath, scanCommand, configFile, am.MultiScanId)
defer func() {
if !cmd.ProcessState.Exited() {
if killProcessError := cmd.Process.Kill(); errorutils.CheckError(killProcessError) != nil {
Expand Down
Loading