Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Scrape the Packit configs and check for URLs used in actions #2212

Closed
4 of 5 tasks
mfocko opened this issue Oct 4, 2023 · 4 comments
Closed
4 of 5 tasks

Scrape the Packit configs and check for URLs used in actions #2212

mfocko opened this issue Oct 4, 2023 · 4 comments
Assignees
@majamassarini @mfocko
Labels
area/other Related to some other area/category than the specified ones. complexity/single-task Regular task, should be done within days. gain/low This doesn't bring that much value to users. impact/low This issue impacts only a few users. kind/internal Doesn't affect users directly, may be e.g. infrastructure, DB related.

Comments

@mfocko
Copy link
Member

mfocko commented Oct 4, 2023 •
edited
Loading

With the move to the MP+ we're now left with the possible issue of upstream / archive URLs being blocked on the firewall. Goal of this issue is to scrape the used configs and find potentially problematic servers that are not allowed.

TODO:

  • Scrape the configs and check for URLs in the actions
  • Potentially do the same for the specfiles as they may include sources kept on various servers
  • Check the connectivity from MP+
    • You can spin up a pod on packit--stg-sandbox and try fetching the sources
    • Or use a tool (though the script in a pod might be simpler)

Related to MP+
@lbarcziova lbarcziova moved this from new to ready-to-refine in Packit Kanban Board Oct 5, 2023
@lbarcziova lbarcziova added complexity/single-task Regular task, should be done within days. impact/low This issue impacts only a few users. area/other Related to some other area/category than the specified ones. gain/low This doesn't bring that much value to users. kind/internal Doesn't affect users directly, may be e.g. infrastructure, DB related. labels Oct 5, 2023
@lbarcziova lbarcziova moved this from ready-to-refine to refined in Packit Kanban Board Oct 5, 2023
@majamassarini majamassarini self-assigned this Oct 5, 2023
@majamassarini majamassarini moved this from refined to in-progress in Packit Kanban Board Oct 5, 2023
@majamassarini
Copy link
Member

I have done

packit_config_checker.py  download-configs
find . -name "*packit*" -exec cat {} \; | awk '/actions/{f=1} /jobs/{f=0;print} f' | grep http > mpp_uri_test.sh

a bit of hand work and I got this script which I ran in our long running worker on stage with no failures:

curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_acd/rubygem-foreman_acd.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/katello/rubygem-foreman_scc_manager/rubygem-foreman_scc_manager.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_snapshot_management/rubygem-foreman_snapshot_management.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/katello/rubygem-katello/rubygem-katello.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; git clone https://github.com/cgwalters/cargo-vendor-filterer.git
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/polkit/raw/rawhide/f/polkit.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/polkit/raw/rawhide/f/polkit.sysusers
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/polkit/raw/rawhide/f/polkit.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/libmks/raw/rawhide/f/libmks.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -s https://src.fedoraproject.org/rpms/crosswords/raw/main/f/crosswords.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -s https://src.fedoraproject.org/rpms/libipuz/raw/main/f/libipuz.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -O https://src.fedoraproject.org/rpms/conmon/raw/rawhide/f/conmon.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -O https://src.fedoraproject.org/rpms/conmon/raw/f37/f/conmon.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -O https://src.fedoraproject.org/rpms/conmon/raw/f36/f/conmon.spe
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/libxcrypt/raw/main/f/libxcrypt.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -O https://src.fedoraproject.org/rpms/conmon/raw/rawhide/f/conmon.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/cri-o/raw/rawhide/f/cri-o.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; git clone https://src.fedoraproject.org/rpms/dfuzzer --depth=1
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; git clone https://src.fedoraproject.org/rpms/elfutils --depth=1
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; git clone -b packit https://pagure.io/meta/fb303.git
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; git clone -b packit https://pagure.io/meta/folly.git
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -s https://src.fedoraproject.org/rpms/netconsd/raw/main/f/netconsd.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -s https://src.fedoraproject.org/rpms/golang-github-facebook-time/raw/main/f/golang-github-facebook-time.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -s https://src.fedoraproject.org/rpms/golang-github-facebookincubator-go2chef/raw/main/f/golang-github-facebookincubator-go2chef.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -s https://src.fedoraproject.org/rpms/python-pystemd/raw/main/f/python-pystemd.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -sSL https://install.python-poetry.org
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/python-fasjson-client/raw/main/f/python-fasjson-client.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -sSL https://install.python-poetry.org
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/fedora-messaging/raw/main/f/fedora-messaging.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -sSL https://install.python-poetry.org
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -o python-flask-oidc.spec https://src.fedoraproject.org/rpms/python-flask-oidc/raw/main/f/python-flask-oidc.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -sSL https://install.python-poetry.org
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/noggin/raw/rawhide/f/noggin.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/noggin/raw/main/f/noggin.service
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/noggin/raw/main/f/noggin.sysconfig
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/noggin/raw/main/f/sources
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -sSL https://install.python-poetry.org
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/python-noggin-messages/raw/main/f/python-noggin-messages.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/python-noggin-messages/raw/main/f/0001-Revert-Include-additional-files-in-the-sdist.patch
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/python-noggin-messages/raw/main/f/README.md
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/python-noggin-messages/raw/main/f/sources
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -sSL https://install.python-poetry.org
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/httpie/raw/rawhide/f/httpie.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/libstoragemgmt/raw/main/f/libstoragemgmt.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/linux-system-roles/raw/rawhide/f/linux-system-roles.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/linux-system-roles/raw/rawhide/f/extrasources.inc
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/linux-system-roles/raw/rawhide/f/ansible-packaging.inc
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/linux-system-roles/raw/rawhide/f/vendoring-prep.inc
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/linux-system-roles/raw/rawhide/f/vendoring-build.inc
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; git clone --branch main --depth 1 https://github.com/dovecot/pigeonhole.git dovecot-pigeonhole
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -s https://src.fedoraproject.org/rpms/python-drgn/raw/main/f/python-drgn.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/p11-kit/raw/rawhide/f/p11-kit.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/p11-kit/raw/rawhide/f/p11-kit-client.service
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://src.fedoraproject.org/rpms/p11-kit/raw/rawhide/f/trust-extract-compat
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; git clone https://gitlab.com/redhat/centos-stream/rpms/systemd.git --depth=1; rm -rf systemd
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; git clone https://github.com/stratis-storage/ci --depth=1
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; git clone https://src.fedoraproject.org/rpms/systemd --depth=1
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl -s https://src.fedoraproject.org/rpms/python-pystemd/raw/main/f/python-pystemd.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/foreman/foreman/foreman.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/foreman/foreman/foreman.logrotate
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/foreman/foreman/foreman.cron.d
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/foreman/foreman/foreman.tmpfiles
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/foreman/foreman-installer/foreman-installer.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman-tasks/rubygem-foreman-tasks.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman-tasks/foreman-tasks.logrotate
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_ansible/rubygem-foreman_ansible.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_bootdisk/rubygem-foreman_bootdisk.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_discovery/rubygem-foreman_discovery.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_leapp/rubygem-foreman_leapp.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/foreman/rubygem-foreman_maintain/rubygem-foreman_maintain.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/foreman/rubygem-foreman_maintain/foreman_maintain.logrotate
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_openscap/rubygem-foreman_openscap.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_plugin_template/rubygem-foreman_plugin_template.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_puppet/rubygem-foreman_puppet.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_remote_execution/rubygem-foreman_remote_execution.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_salt/rubygem-foreman_salt.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_templates/rubygem-foreman_templates.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-foreman_webhooks/rubygem-foreman_webhooks.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/foreman/rubygem-hammer_cli/rubygem-hammer_cli.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/foreman/rubygem-hammer_cli_foreman/rubygem-hammer_cli_foreman.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-hammer_cli_foreman_ansible/rubygem-hammer_cli_foreman_ansible.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; curl https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/plugins/rubygem-smart_proxy_ansible/rubygem-smart_proxy_ansible.spec
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi; git clone https://src.fedoraproject.org/rpms/util-linux.git --depth=1
if [ $? == 0 ]; then echo PASS; else echo FAILED; fi

@mfocko do we want to check also the specfile(s) even though it seems not to be problems accessing http uri?

@majamassarini majamassarini moved this from in-progress to in-review in Packit Kanban Board Oct 6, 2023
@mfocko
Copy link
Member Author

mfocko commented Oct 11, 2023

@majamassarini I'll have a look today, I must've missed the notification /o\

@mfocko
Copy link
Member Author

mfocko commented Oct 12, 2023

Thanks a lot, I've went through the specfiles too and found these domains that hold sources and we cannot access them:

  • go.dev
  • pigeonhole.dovecot.org
  • dovecot.org

I'll create a ticket for the IT to enable those.

@mfocko mfocko closed this as completed Oct 12, 2023
@github-project-automation github-project-automation bot moved this from in-review to done in Packit Kanban Board Oct 12, 2023
@mfocko mfocko self-assigned this Oct 12, 2023
@majamassarini
Copy link
Member

majamassarini commented Oct 12, 2023 •
edited
Loading

Thanks a lot, I've went through the specfiles too and found these domains that hold sources and we cannot access them:

Oh thank you, I didn't mean you to do this 😅 Thank a lot to you for having done it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@majamassarini majamassarini

@mfocko mfocko

Labels
area/other Related to some other area/category than the specified ones. complexity/single-task Regular task, should be done within days. gain/low This doesn't bring that much value to users. impact/low This issue impacts only a few users. kind/internal Doesn't affect users directly, may be e.g. infrastructure, DB related.
Projects
Packit Kanban Board
Archived in project
Milestone
No milestone
Development

No branches or pull requests
3 participants
@majamassarini @mfocko @lbarcziova