Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fully defined apt repository package checking #2554

Merged
merged 5 commits into from
Feb 27, 2024
Merged

Fully defined apt repository package checking #2554

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9b55dfe
api: add `debug` function to `echo` input when `pi_apps_debug` == true
theofficialgman Feb 25, 2024
01cf9a4
api: add `anything_installed_from_uri_suite_component` to fully defin…
theofficialgman Feb 24, 2024
7a6934b
api: refactor `remove_repofile_if_unused` to use `anything_installed_…
theofficialgman Feb 24, 2024
782b893
api: `anything_installed_from_uri_suite_component` simplify function
theofficialgman Feb 25, 2024
64c8fdd
api: remove `anything_installed_from_repo` function
theofficialgman Feb 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
152 changes: 107 additions & 45 deletions api
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,10 @@ status_green() { #announce the success of a major action
echo -e "\e[92m$1\e[0m" 1>&2
}

debug() { #an echo command that only runs when debug mode is on
[ "$pi_apps_debug" = true ] && echo "$1"
}

Botspot marked this conversation as resolved.
Show resolved Hide resolved
generate_logo() { #display colorized Pi-Apps logo in terminal
#ANSI color codes: https://misc.flogisoft.com/bash/tip_colors_and_formatting
#Search for unicode characters: https://unicode-search.net - search for "block" and "quadrant"
Expand Down Expand Up @@ -179,45 +183,43 @@ package_is_new_enough() { #check if the $1 package has an available version grea
fi
}

anything_installed_from_repo() { #Given an apt repository URL, determine if any packages from it are currently installed
[ -z "$1" ] && error "anything_installed_from_repo: A repository URL must be specified."
anything_installed_from_uri_suite_component() { #Given an apt repository uri, suite, and component, determine if any packages from it are currently installed
local uri="$1"
local suite="$2"
component="$3" #can be left blank
[ -z "$uri" ] && error "anything_installed_from_uri_suite_component: A repository uri must be specified."
[ -z "$suite" ] && error "anything_installed_from_uri_suite_component: A repository suite must be specified."

#user input repo-url. Remove 'https://', and translate '/' to '_' to conform to apt file-naming standard, with trailing _ to ensure full matches
local url="$(echo "$1" | sed 's+.*://++g' | tr '/' '_')_"
#find part of path to apt list file(s) to search for
if [ -z "$component" ]; then
local filepath="/var/lib/apt/lists/$(echo "$1" | sed 's+.*://++g' | sed "s,/$,," | tr '/' '_')_$(echo "$suite" | sed "s,/$,," | tr '/' '_')_"
else
local filepath="/var/lib/apt/lists/$(echo "$1" | sed 's+.*://++g' | sed "s,/$,," | tr '/' '_')_dists_$(echo "$suite" | sed "s,/$,," | tr '/' '_')_$(echo "$component" | sed "s,/$,," | tr '/' '_')_"
fi
debug $filepath

#find all package-lists pertaining to the url
local repofiles="$(ls /var/lib/apt/lists/*_Packages | grep -F "$url")"
#find all relevant package-lists
local repofiles="$(ls ${filepath}*_Packages)"
debug "$repofiles"

#for every repo-file, check if any of them have an installed file
local found=0
local IFS=$'\n'
local repofile
local installed_packages="$(grep -xF 'Status: install ok installed' /var/lib/dpkg/status -B 2 | grep '^Package: ' | sed 's/^Package: //g' | sort)"
for repofile in $repofiles ;do
#search the repo-file for installed packages

grep '^Package' "$repofile" | awk '{print $2}' | while read -r package ;do
if package_installed "$package" ;then
#this package is installed; check if the version available on this repo is the current version (prevents false positives from backports repos)
if [ "$(sed -n "/^Package: ${package}$/,/Version:/p" "$repofile" | grep '^Version: ' | awk '{print $2}')" == "$(package_installed_version "$package")" ];then
echo "Package installed: $package"
exit 1
fi
fi
done #if exit code is 1, search was successful. If exit code is 0, no packages from the repo were installed.

found=$?
local packages_in_repo="$(grep '^Package: ' "$repofile" | awk '{print $2}' | sort)"
local apt_cache_policy_output="$(echo "$packages_in_repo" | list_intersect "$installed_packages" | tr '\n' ' ' | xargs -r apt-cache policy)"
Copy link
Owner

@Botspot Botspot Feb 26, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With tr used here, it seems to me like the output from this would always be a single line, meaning all matching packages would be passed to apt-cache in one run.
So it seems like the only use for xargs is shorthand to avoid running apt-cache if no packages from the repo are installed?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

apt-cache policy needs the input as individual arguments. it also does not read from stdin. these are the reasons why xargs is used.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I explained poorly.
If apt-cache is only being run once, then something like this should work if I understand this correctly

local apt_cache_policy_output="$(apt-cache policy $(echo "$packages_in_repo" | list_intersect "$installed_packages" | tr '\n' ' '))"

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No I correctly interpreted what you said.

apt-cache policy needs the input as individual arguments.

What you just sent puts all output as one input argument I think. Can't check now.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No quotes around the inner $() will make each package into a separate argument.
If nothing else, this is simply something useful to know. I'm not necessarily requesting any changes here.


if [ $found == 1 ];then
break
#check if any installed packages also found on this repo are actually installed from this repo
if [ -z "$component" ]; then
echo "$apt_cache_policy_output" | grep -B1 "$(echo "$uri" | sed 's+.*://++g' | sed "s,/$,,") $suite" | awk '{print $1}' | grep -Fq '***' && return 0
else
echo "$apt_cache_policy_output" | grep -B1 "$(echo "$uri" | sed 's+.*://++g' | sed "s,/$,,") $suite/$component" | awk '{print $1}' | grep -Fq '***' && return 0
fi
done

#return an exit code
if [ $found == 1 ];then
return 0
else
return 1
fi
return 1
}

remove_repofile_if_unused() { #Given a sources.list.d file, delete it if nothing from that repository is currently installed. Deletion skipped if $2 is 'test'
Expand All @@ -227,34 +229,94 @@ remove_repofile_if_unused() { #Given a sources.list.d file, delete it if nothing
[ -z "$file" ] && error "remove_repo_if_unused: no sources.list.d file specified!"
#return now if the list file does not exist
[ -f "$file" ] || return 0

#set default to not in use
local in_use=0

if [ "${file##*.}" == "list" ]; then
#determine what repo-urls are in the file: include dist information. Example value: deb.debian.org_debian_dists_bookworm
local urls="$(cat "$file" | grep -v '^#' | tr ' ' '\n' | grep '://' -A1 | tr '\n' ' ' | sed 's/ -- /\n/g ; s/ $/\n/g ; s/ /_dists_/g')"
# determine what uri, suite, and components are in a file
local lines="$(cat "$file" | grep "^deb " | sed 's/^deb // ; s/\[.*\]//')"
local IFS=$'\n'
for line in $lines ;do
local uri="$(echo "$line" | awk '{print $1}')"
local suite="$(echo "$line" | awk '{print $2}')"
local components="$(echo "$line" | awk '{$1=$2=""; print $0}')"
local IFS=' '
if [ -z "$components" ]; then
debug "$uri $suite"
if anything_installed_from_uri_suite_component "$uri" "$suite";then
in_use=1
break 1
fi
else
for component in $components ;do
debug "$uri $suite $component"
if anything_installed_from_uri_suite_component "$uri" "$suite" "$component";then
in_use=1
break 2
fi
done
fi
local IFS=$'\n'
done
elif [ "${file##*.}" == "sources" ]; then
#determine what repo-urls are in the file
local urls="$(cat "$file" | grep -v '^#' | grep '^URIs: ' | sed 's/URIs: //g')"
#find empty lines (empty line, line with all spaces, or line with all tabs) in the file that separate stanzas. empty lines are not allowed between fields within a stanza
#https://manpages.ubuntu.com/manpages/jammy/en/man5/deb822.5.html
local empty_lines="$(grep -P -n '^$|^ +$|^\t+$' "$file" | awk '{print $1}' | sed 's/:*//g')"
#get number of lines in file
local num_lines=$(wc -l "$file" | awk '{print $1}')

#always add last line to empty lines
if [ -z "$empty_lines" ]; then
empty_lines="$num_lines"
else
empty_lines+=$'\n'"$num_lines"
fi
debug "$empty_lines"

#parse each stanza, starting at line 1
local IFS=$'\n'
local line_start=1
for line_end in $empty_lines ;do
# if Enabled: no, continue to next loop iteration
sed -n "$line_start","$line_end"p "$file" | grep -q '^Enabled: no' && continue
# determine what uri, suite, and components are in a file
#each stanza can only have one matching section. if there are multiple matches the last one is used
#case should be ignored for fields
local uris="$(sed -n "$line_start","$line_end"p "$file" | grep -v '^#' | grep -i '^URIs:' | sed 's/URIs://Ig' | awk '{$1=$1};1' | tail -1)"
local suites="$(sed -n "$line_start","$line_end"p "$file" | grep -v '^#' | grep -i '^Suites:' | sed 's/Suites://Ig' | awk '{$1=$1};1' | tail -1)"
local components="$(sed -n "$line_start","$line_end"p "$file" | grep -v '^#' | grep -i '^Components:' | sed 's/Components://Ig' | awk '{$1=$1};1' | tail -1)"
local IFS=' '
for uri in $uris ;do
for suite in $suites ;do
if [ -z "$components" ]; then
debug "$uri $suite"
if anything_installed_from_uri_suite_component "$uri" "$suite";then
in_use=1
break 3
fi
else
for component in $components ;do
debug "$uri $suite $component"
if anything_installed_from_uri_suite_component "$uri" "$suite" "$component";then
in_use=1
break 4
fi
done
fi
done
done
local IFS=$'\n'
line_start="$line_end"
done
else
error "$file was not of apt list or sources type"
fi

#there could be multiple urls in one file. Check each url and set the in_use variable to 1 if any packages are found
local IFS=$'\n'
local in_use=0
local url
for url in $urls ;do
if anything_installed_from_repo "$url" >/dev/null;then
in_use=1
break
fi
done

if [ "$testmode" == test ] && [ "$in_use" == 0 ];then
echo "The given repository is not in use and can be deleted:"$'\n'"$file" 1>&2
elif [ "$testmode" == test ];then
#explain what package is installed from this repo
echo "At least this package is preventing the repo from being removed (there may be more)"
anything_installed_from_repo "$url" #use value of $url from earlier
echo "At least one package is preventing the repo from being removed"
elif [ "$in_use" == 0 ];then
status "Removing the $(basename "$file" | sed 's/.list$//g' | sed 's/.sources$//g') repo as it is not being used"
sudo rm -f "$file"
Expand Down
2 changes: 1 addition & 1 deletion apps/Brave/install-64
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ sudo curl -fsSLo /usr/share/keyrings/brave-browser-archive-keyring.gpg https://b

(install_packages brave-browser)
if [ $? != 0 ]; then
if ! anything_installed_from_repo "https://brave-browser-apt-release.s3.brave.com/" ; then
if ! anything_installed_from_uri_suite_component "https://brave-browser-apt-release.s3.brave.com/" stable main ; then
# nothing installed from repo, this check is to prevent removing repos which other pi-apps scripts or the user have used successfully
# safe to remove
sudo rm -f /etc/apt/sources.list.d/brave-browser-release.list /usr/share/keyrings/brave-browser-archive-keyring.gpg
Expand Down
Loading