feat(python): add pypa support #96

AndreyLevchenko · 2021-08-11T10:58:16Z

No description provided.

add test

fix typo

knqyf263 · 2021-08-23T12:09:44Z

pypa/pypa.go

+}
+type option func(*options)
+
+type Pypa struct {


Suggested change

type Pypa struct {

type PyPA struct {

According to their website, PyPA is correct.
https://www.pypa.io/en/latest/

knqyf263 · 2021-08-23T12:10:26Z

pypa/pypa.go

+type options struct {
+	url   string
+	dir   string
+	retry int


retry seems to be unused.

knqyf263 · 2021-08-23T12:12:15Z

pypa/pypa.go

+	bar := pb.StartNew(len(yamlFiles))
+
+	for _, file := range yamlFiles {
+		data, err := ioutil.ReadFile(file)


ioutil is deprecated

Suggested change

data, err := ioutil.ReadFile(file)

data, err := os.ReadFile(file)

knqyf263 · 2021-08-23T12:18:22Z

pypa/pypa_test.go

+			}
+
+			dir := t.TempDir()
+			c := NewPypa(WithURL(ts.URL+"/"+tt.inputArchive), WithDir(filepath.Join(dir)), WithRetry(0))


You have to pass afero.NewMemMapFs() and need to use the filesystem below instead of os.ReadDir and os.ReadFile.

based on our chat we decided not to do it, because of go-getterincompatibility

knqyf263 · 2021-08-23T12:18:37Z

pypa/pypa_test.go

@@ -0,0 +1,78 @@
+package pypa


Suggested change

package pypa

package pypa_test

knqyf263 · 2021-08-23T12:19:48Z

pypa/pypa_test.go

+			} else {
+				require.NoError(t, err)
+			}


Suggested change

} else {

require.NoError(t, err)

}

}

require.NoError(t, err)

knqyf263 · 2021-08-23T12:20:58Z

main.go

+	case "pypa":
+		p := pypa.NewPypa()
+		if err := p.Update(); err != nil {
+			return xerrors.Errorf("error in Pypa update: %w", err)


Suggested change

return xerrors.Errorf("error in Pypa update: %w", err)

return xerrors.Errorf("error in PyPA update: %w", err)

knqyf263 · 2021-08-23T12:21:08Z

main.go

+		if err := p.Update(); err != nil {
+			return xerrors.Errorf("error in Pypa update: %w", err)
+		}
+		commitMsg = "Pypa Security Advisories"


Suggested change

commitMsg = "Pypa Security Advisories"

commitMsg = "PyPA Security Advisories"

knqyf263 · 2021-08-23T12:21:47Z

pypa/pypa.go

+		return xerrors.Errorf("failed to download %s: %w", pypa.opts.url, err)
+	}
+
+	vulnDir := filepath.Join(dir, "advisory-db-main/vulns")


Suggested change

vulnDir := filepath.Join(dir, "advisory-db-main/vulns")

vulnDir := filepath.Join(dir, "advisory-db-main", "vulns")

knqyf263 · 2021-08-23T12:23:26Z

pypa/pypa.go

+			return xerrors.Errorf("unable to parse yaml %s: %w", file, err)
+		}
+
+		if err := utils.WriteJSON(pypa.AppFs, pypa.opts.dir, fmt.Sprintf("%s.json", osv.Id), osv); err != nil {


We should keep the directory structure as much as possible. It means we will have package dirs.
https://github.com/pypa/advisory-db/tree/main/vulns

patryk4815 · 2021-09-02T14:55:02Z

pypa/pypa.go

+
+const (
+	pypaDir            = "pypa"
+	securityTrackerURL = "https://github.com/pypa/advisory-db/archive/refs/heads/main.zip"


why not using https://osv-vulnerabilities.storage.googleapis.com/PyPI/all.zip ?
https://github.com/google/osv/blob/master/README.md#current-data-sources

@AndreyLevchenko Could you check if we can use the above URL?

@knqyf263 I think I can use the url above, but I need to understand benefits better. For now it looks pretty same to me.

The benefit is we can have a single implementation for some languages (Python, Go and Rust).

patryk4815 · 2021-09-03T00:44:34Z

I think we should change this PR and #83
Into one implementation for handling osv as datasource (like ghsa target):
https://github.com/google/osv/blob/master/README.md#current-data-sources
They support multiple source (python/go/etc) with same schema.

knqyf263 · 2021-09-04T05:52:34Z

@patryk4815 It sounds like a nice idea. Go data source in OSV used to have an issue (golang/vulndb#1) and we needed to parse the repository ourselves. But after having a quick look, I found they might change API. Let me look into OSV carefully. If we can implement osv, it would be better. Thanks.

patryk4815 · 2021-09-05T21:52:41Z

@knqyf263 Maybe we need wait for "purl" support - google/osv.dev#64

CLAassistant · 2021-09-14T08:44:53Z

All committers have signed the CLA.

westonsteimel · 2021-10-13T00:18:00Z

@knqyf263, the OSV schema specification should at least be stable now per https://ossf.github.io/osv-schema/#status---2021-09-08. Let me know if there are any issues you're experiencing from the OSV API side and I can try to take a look. I've put quite a significant effort into triage for the PyPA Advisory DB, so definitely keen to help where I can to see it used.

knqyf263 · 2021-12-19T11:33:07Z

@westonsteimel Thanks! It looks like PyPA Advisory DB works well for us. We have a problem with Go.

knqyf263 · 2021-12-19T11:33:25Z

In favor of #113

AndreyLevchenko added 3 commits August 11, 2021 16:55

feat(python): add pypa support

f4e02b8

feat(python): add pypa support

a33d1f9

add test

feat(python): add pypa support

8a06a66

fix typo

knqyf263 requested changes Aug 23, 2021

View reviewed changes

AndreyLevchenko added 2 commits August 26, 2021 13:43

fix initial set of PR notes

95ea5cc

add package based subfolders

cfd2587

AndreyLevchenko marked this pull request as ready for review August 26, 2021 10:31

patryk4815 reviewed Sep 2, 2021

View reviewed changes

knqyf263 mentioned this pull request Oct 10, 2021

Add trivy image --skip-analyzers option aquasecurity/trivy#1236

Closed

knqyf263 closed this Dec 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(python): add pypa support #96

feat(python): add pypa support #96

AndreyLevchenko commented Aug 11, 2021

knqyf263 Aug 23, 2021

AndreyLevchenko Aug 26, 2021

knqyf263 Aug 23, 2021

AndreyLevchenko Aug 26, 2021

knqyf263 Aug 23, 2021

AndreyLevchenko Aug 26, 2021

knqyf263 Aug 23, 2021

AndreyLevchenko Aug 26, 2021

knqyf263 Aug 23, 2021

AndreyLevchenko Aug 26, 2021

knqyf263 Aug 23, 2021

AndreyLevchenko Aug 26, 2021

knqyf263 Aug 23, 2021

AndreyLevchenko Aug 26, 2021

knqyf263 Aug 23, 2021

AndreyLevchenko Aug 26, 2021

knqyf263 Aug 23, 2021

AndreyLevchenko Aug 26, 2021

knqyf263 Aug 23, 2021

AndreyLevchenko Aug 26, 2021

patryk4815 Sep 2, 2021

knqyf263 Sep 2, 2021

AndreyLevchenko Sep 7, 2021

knqyf263 Sep 7, 2021

patryk4815 commented Sep 3, 2021

knqyf263 commented Sep 4, 2021

patryk4815 commented Sep 5, 2021

CLAassistant commented Sep 14, 2021 •

edited

Loading

westonsteimel commented Oct 13, 2021

knqyf263 commented Dec 19, 2021

knqyf263 commented Dec 19, 2021

	data, err := ioutil.ReadFile(file)
	data, err := os.ReadFile(file)

	return xerrors.Errorf("error in Pypa update: %w", err)
	return xerrors.Errorf("error in PyPA update: %w", err)

	commitMsg = "Pypa Security Advisories"
	commitMsg = "PyPA Security Advisories"

	vulnDir := filepath.Join(dir, "advisory-db-main/vulns")
	vulnDir := filepath.Join(dir, "advisory-db-main", "vulns")

feat(python): add pypa support #96

feat(python): add pypa support #96

Conversation

AndreyLevchenko commented Aug 11, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patryk4815 commented Sep 3, 2021

knqyf263 commented Sep 4, 2021

patryk4815 commented Sep 5, 2021

CLAassistant commented Sep 14, 2021 • edited Loading

westonsteimel commented Oct 13, 2021

knqyf263 commented Dec 19, 2021

knqyf263 commented Dec 19, 2021

CLAassistant commented Sep 14, 2021 •

edited

Loading