[ANE-949] Deep / Transitive dependencies for setuptool projects

[jagonalez]

Overview

PR adds support for determining deep / transitive dependencies for setuptool python projects (seutp.py, setup.cfg, requirements.txt). This is done by shelling out to python -m pip list and python -m pip show. pip list will give us a list of installed packages, where pip show will give us package dependencies. If neither python nor pip are installed then we fall back to current implements.

For req*.txt - only transitive dependencies for packages found in a req*.txt file will be reported.
For setup.py/setup.cfg, we attempt to parse the name, if a name is found we attempt to find a package installed with the same name and add transitives dependencies for those dependencies. If we cannot parse the name, or do not find a package installed with the same name, we fall back to only reporting the direct dependencies parsed in the config file.

Acceptance criteria

• transitive dependencies will be reported for python packages which have been installed in an environment
• if no packages are installed then only direct dependencies are reported

Testing plan

• create a requirements.txt file:

urllib3===2.0.7
requests

• run fossa analyze within the directory with the requirements.txt file, results should look like this:

{"projects":[{"graph":{"assocs":[],"deps":[{"locations":[],"name":"requests","tags":{},"type":"PipType","version":null},{"locations":[],"name":"urllib3","tags":{},"type":"PipType","version":{"type":"EQUAL","value":"2.0.7"}}],"direct":[0,1]},"path":"/home/jeremy/dev/fossa-cli-python-test/","type":"setuptools"}],"sourceUnits":[{"AdditionalDependencyData":null,"Build":{"Artifact":"default","Dependencies":[{"imports":[],"locator":"pip+requests$"},{"imports":[],"locator":"pip+urllib3$2.0.7"}],"Imports":["pip+requests$","pip+urllib3$2.0.7"],"Succeeded":true},"Data":null,"Files":null,"GraphBreadth":"partial","Info":null,"Manifest":"/home/jeremy/dev/fossa-cli-python-test/","Name":"/home/jeremy/dev/fossa-cli-python-test/","OriginPaths":["requirements.txt"],"Title":null,"Type":"setuptools"}]}

• create a setup.py file within the same directory:

from setuptools import setup

setup(
        name='foo',
        version='1',
        description='whatever',
        install_requires=['bar'],
)

• run fossa analyze -o in the same directory, results should look like:

{"projects":[{"graph":{"assocs":[],"deps":[{"locations":[],"name":"bar","tags":{},"type":"PipType","version":null},{"locations":[],"name":"requests","tags":{},"type":"PipType","version":null},{"locations":[],"name":"urllib3","tags":{},"type":"PipType","version":{"type":"EQUAL","value":"2.0.7"}}],"direct":[0,1,2]},"path":"/home/jeremy/dev/fossa-cli-python-test/","type":"setuptools"}],"sourceUnits":[{"AdditionalDependencyData":null,"Build":{"Artifact":"default","Dependencies":[{"imports":[],"locator":"pip+bar$"},{"imports":[],"locator":"pip+requests$"},{"imports":[],"locator":"pip+urllib3$2.0.7"}],"Imports":["pip+bar$","pip+requests$","pip+urllib3$2.0.7"],"Succeeded":true},"Data":null,"Files":null,"GraphBreadth":"partial","Info":null,"Manifest":"/home/jeremy/dev/fossa-cli-python-test/","Name":"/home/jeremy/dev/fossa-cli-python-test/","OriginPaths":["setup.py","requirements.txt"],"Title":null,"Type":"setuptools"}]}

• create a new python env: python -m venv ~/envs/test-fossa-cli, make sure you are in the python env if you need to run source ~/envs/test-fossa-cli/activate
• install the deps: pip install -r requirements.txt
• run fossa analyze -o within the directory with the requirements.txt file, results should simliar:

{"projects":[{"graph":{"assocs":[[4,[1,2,3,5]]],"deps":[{"locations":[],"name":"bar","tags":{},"type":"PipType","version":null},{"locations":[],"name":"certifi","tags":{},"type":"PipType","version":{"type":"EQUAL","value":"2023.11.17"}},{"locations":[],"name":"charset-normalizer","tags":{},"type":"PipType","version":{"type":"EQUAL","value":"3.3.2"}},{"locations":[],"name":"idna","tags":{},"type":"PipType","version":{"type":"EQUAL","value":"3.4"}},{"locations":[],"name":"requests","tags":{},"type":"PipType","version":null},{"locations":[],"name":"urllib3","tags":{},"type":"PipType","version":{"type":"EQUAL","value":"2.0.7"}}],"direct":[0,4,5]},"path":"/home/jeremy/dev/fossa-cli-python-test/","type":"setuptools"}],"sourceUnits":[{"AdditionalDependencyData":null,"Build":{"Artifact":"default","Dependencies":[{"imports":[],"locator":"pip+bar$"},{"imports":[],"locator":"pip+certifi$2023.11.17"},{"imports":[],"locator":"pip+charset-normalizer$3.3.2"},{"imports":[],"locator":"pip+idna$3.4"},{"imports":["pip+certifi$2023.11.17","pip+charset-normalizer$3.3.2","pip+idna$3.4","pip+urllib3$2.0.7"],"locator":"pip+requests$"},{"imports":[],"locator":"pip+urllib3$2.0.7"}],"Imports":["pip+bar$","pip+requests$","pip+urllib3$2.0.7"],"Succeeded":true},"Data":null,"Files":null,"GraphBreadth":"partial","Info":null,"Manifest":"/home/jeremy/dev/fossa-cli-python-test/","Name":"/home/jeremy/dev/fossa-cli-python-test/","OriginPaths":["setup.py","requirements.txt"],"Title":null,"Type":"setuptools"}]}

• run pip install .,
• run fossa analyze -o within the directory, results should be similar:

{"projects":[{"graph":{"assocs":[],"deps":[{"locations":[],"name":"bar","tags":{},"type":"PipType","version":null}],"direct":[0]},"path":"/home/jeremy/dev/fossa-cli-python-test/foo.egg-info/","type":"setuptools"},{"graph":{"assocs":[[4,[1,2,3,5]]],"deps":[{"locations":[],"name":"bar","tags":{},"type":"PipType","version":{"type":"EQUAL","value":"0.2.1"}},{"locations":[],"name":"certifi","tags":{},"type":"PipType","version":{"type":"EQUAL","value":"2023.11.17"}},{"locations":[],"name":"charset-normalizer","tags":{},"type":"PipType","version":{"type":"EQUAL","value":"3.3.2"}},{"locations":[],"name":"idna","tags":{},"type":"PipType","version":{"type":"EQUAL","value":"3.4"}},{"locations":[],"name":"requests","tags":{},"type":"PipType","version":null},{"locations":[],"name":"urllib3","tags":{},"type":"PipType","version":{"type":"EQUAL","value":"2.0.7"}}],"direct":[0,4,5]},"path":"/home/jeremy/dev/fossa-cli-python-test/","type":"setuptools"}],"sourceUnits":[{"AdditionalDependencyData":null,"Build":{"Artifact":"default","Dependencies":[{"imports":[],"locator":"pip+bar$"}],"Imports":["pip+bar$"],"Succeeded":true},"Data":null,"Files":null,"GraphBreadth":"partial","Info":null,"Manifest":"/home/jeremy/dev/fossa-cli-python-test/foo.egg-info/","Name":"/home/jeremy/dev/fossa-cli-python-test/foo.egg-info/","OriginPaths":["foo.egg-info/requires.txt"],"Title":null,"Type":"setuptools"},{"AdditionalDependencyData":null,"Build":{"Artifact":"default","Dependencies":[{"imports":[],"locator":"pip+bar$0.2.1"},{"imports":[],"locator":"pip+certifi$2023.11.17"},{"imports":[],"locator":"pip+charset-normalizer$3.3.2"},{"imports":[],"locator":"pip+idna$3.4"},{"imports":["pip+certifi$2023.11.17","pip+charset-normalizer$3.3.2","pip+idna$3.4","pip+urllib3$2.0.7"],"locator":"pip+requests$"},{"imports":[],"locator":"pip+urllib3$2.0.7"}],"Imports":["pip+bar$0.2.1","pip+requests$","pip+urllib3$2.0.7"],"Succeeded":true},"Data":null,"Files":null,"GraphBreadth":"partial","Info":null,"Manifest":"/home/jeremy/dev/fossa-cli-python-test/","Name":"/home/jeremy/dev/fossa-cli-python-test/","OriginPaths":["setup.py","requirements.txt"],"Title":null,"Type":"setuptools"}]}

Risks

Metrics

Is this change something that can or should be tracked? If so, can we do it today? And how? If its easy, do it

References

• ANE-949: Detect deep/transitive dependencies in Python with setup.py or requirements.txt

Checklist

• I added tests for this PR's change (or explained in the PR description why tests don't make sense).
• If this PR introduced a user-visible change, I added documentation into docs/.
• If this change is externally visible, I updated Changelog.md. If this PR did not mark a release, I added my changes into an # Unreleased section at the top.
• If I made changes to .fossa.yml or fossa-deps.{json.yml}, I updated docs/references/files/*.schema.json. You may also need to update these if you have added/removed new dependency type (e.g. pip) or analysis target type (e.g. poetry).
• If I made changes to a subcommand's options, I updated docs/references/subcommands/<subcommand>.md.

[meghfossa]

Added some comments as I'm working through it - would be able to wrap up review later today.

[meghfossa]

• Gr8 work, this will have noticeable impact for our python users!
• One more change (mostly on discovery end)!

Other bits,

• Add integration test (since this will be heavily used by our python users)

-- integration-test/Analysis/SetuptoolsSpec.hs

-- Here is templated fixture, it may be easier to simply 
-- perform fixture/test without below pattern, if you are
-- running into issue with nixEnv, and buildCommand

-- To test this locally:
-- cabal test integration-tests --test-show-details=streaming --test-option=--format=checks --test-option=--match --test-option="wildCardForTest"
pythonDynamic :: AnalysisTestFixture (Setuptools.SetuptoolsProject)
pythonDynamic =
  AnalysisTestFixture
    "pythonDynamic" 
    Setuptools.discover
    LocalEnvironment -- You may have to use nix env and build cmd
    Nothing -- Build command pip install, and virtual env, etc, 
    $ FixtureArtifact
      "https://github.com/pallets/flask/archive/refs/tags/2.0.2.tar.gz"
      [reldir|python/setuptools/flask/|]
      [reldir|flask-2.0.2/|]

spec :: Spec
spec = do
  testSuiteDepResultSummary pythonDynamic SetuptoolsProjectType $
  DependencyResultsSummary 
     numDeps 
     numDirectDeps 
     numEdges 
     numManifestFiles 
     Complete

• Update changelog before merge!
• [Optional] Include example project in example-project repo

[meghfossa]