Fixes 4822: rename domains with cs- prefix #865

jlsherrill · 2024-10-25T01:23:41Z

Summary

Changes template model to no longer contain RepositoryConfigurations directly, but instead TemplateRepositoryConfigurations, this was due to an observed bug with gorm. Strangely this bug was not always reproducible, but it was observed that querying Templates with the default scope resulted in RepositoryConfigurations included that had been removed from the template.

This moves all the jobs to one command ./cmd/jobs/main.go that takes the name of the job as an argument.

Deletes the repair_latest and cleanup_versions jobs

Adds a new job 'rename_domains' that renames the red hat domain to 'cs-redhat' if its not already that. Renames all other domains to cs-PREVIOUS_NAME if it doesn't already start with 'cs-'.

The rename process involves:

loading the old name from the db
updating the paths for all the templates in candlepin
updating the content overrides for all the templates in candlepin TODO
updating the domain name in pulp
updating our own domains table

5 is done last so that if the process errors at any point, it can be re-run from the start for that domain simply by re-running the job.

Testing steps

On main, revert this commit: ebdbba4
run:

make repos-import-rhel9
go run cmd/external-repos/main.go nightly-jobs

start the server and let the RHEL repos snapshot
3. Create at least 1 custom repo with snapshotting enabled
4. Create a content template, assign rhel9 and custom repo(s)
5. follow https://github.com/content-services/content-sources-backend/blob/main/docs/register_client.md to register a client
6. confirm that the client can install content form the redhat repos and custom repos
7. rename the domains: go run cmd/jobs/main.go rename_domains
8. On the client run subscription-manager refresh && subscription-manager repo-override
9. verify that you can still install content from the red hat and custom repos
10. confirm that the templates urls now point to cs-DOMAIN & cs-redhat
11. confirm that the repo configs in the webUI for snapshots still work
12. If you still see the old domains in the redhat.repo file, then try re-registering

jlsherrill · 2024-10-25T01:30:18Z

https://issues.redhat.com/browse/HMS-4822

xbhouse · 2024-11-05T16:18:44Z

for existing repo snapshots, after running the rename-domains job i see the old domains when fetching the repos and so the urls point to a 404. also see the old domains in the config.repos from existing repo snapshots. i'm guessing it is expected to have to trigger new snapshots before we'll see the new domains in these places?

xbhouse · 2024-11-05T17:13:37Z

pkg/dao/domains.go

@@ -32,8 +41,13 @@ func (dDao domainDaoImpl) FetchOrCreateDomain(ctx context.Context, orgId string)
 }

 func (dDao domainDaoImpl) Create(ctx context.Context, orgId string) (string, error) {
+	name := fmt.Sprintf("cs-%v", random.New().String(10, random.Hex))


didn't notice in the previous PR but old custom domains (prefixed with cs-) have a length of 11 and new ones have a length of 13. my guess is that they were originally set to the same length as an org id? just wanted to point this out, do you think it would cause any issues?

i'm assuming it shouldn't matter since the new red hat domain is also a different length 😄

yeah, i think it was just chosen to be small, but not too small. The actual size doesn't matter too much

pkg/jobs/rename_domains.go

xbhouse · 2024-11-05T17:28:06Z

everything works as described on the client 🎉 just a couple questions above and a nitpick, overall nice job!!

jlsherrill · 2024-11-05T19:24:15Z

for existing repo snapshots, after running the rename-domains job i see the old domains when fetching the repos and so the urls point to a 404. also see the old domains in the config.repos from existing repo snapshots. i'm guessing it is expected to have to trigger new snapshots before we'll see the new domains in these places?

Hrmmmm this sounds like something didn't go as planned, I'll try to reproduce.

xbhouse · 2024-11-05T20:03:51Z

for existing repo snapshots, after running the rename-domains job i see the old domains when fetching the repos and so the urls point to a 404. also see the old domains in the config.repos from existing repo snapshots. i'm guessing it is expected to have to trigger new snapshots before we'll see the new domains in these places?

Hrmmmm this sounds like something didn't go as planned, I'll try to reproduce.

ah, also i'm seeing this error from the delete-repository-snapshots task when deleting those existing custom repos:

"error reading task: 404 Not Found: \n<!doctype html>\n<html lang=\"en\">\n<head>\n  <title>Not Found</title>\n</head>\n<body>\n  <h1>Not Found</h1><p>The requested resource was not found on this server.</p>\n</body>\n</html>\n"

to reproduce these things:

revert this commit in main
import a red hat repo and add a custom repo and wait for both to snapshot
checkout this pr and run rename_domains
list repos (snapshot urls show old domain)
copy the config.repo output in the UI (baseurl shows old domain)
delete the custom repo

rverdile · 2024-11-06T17:34:20Z

pkg/dao/interfaces.go

-func SetupGormTableOrFail(db *gorm.DB) {
-	err := db.SetupJoinTable(models.Template{}, "RepositoryConfigurations", models.TemplateRepositoryConfiguration{})
-	if err != nil {
-		log.Logger.Fatal().Err(err).Msg("error setting up join table for templates_repository_configurations")
-	}
-}


Was this causing that bug? Will soft delete still work without it?

When i made the change here: 3544d27#diff-6daee329a68afa6ef518701e49dba065bb5372d88bec90794533ff924bf576d5R27 to link templates to TemplateRepositoryConfiguration directly, it complained. My guess is this might have been why it wasn't working before? (it was trying to link through that table to RepositoryConfigurations directly).

what sort of behavior were you seeing? Just the template_repository_configuration entry being deleted entirely instead of soft deleted?

I suspect the change to the model may have made it okay to be removed. I noticed there was a comment above this function that said "we need this for soft delete to work", so I wanted to make sure you caught that.

You can see in the gorm docs, the models are arranged like what we previously had: https://gorm.io/docs/many_to_many.html#Customize-JoinTable

i did a manual test by creating a template with 2 repos, I think ran the api server without any workers. I then updated the template to only have 1 repo. Finally I checked the db:

content=# select template_uuid, deleted_at from templates_repository_configurations ; template_uuid | deleted_at --------------------------------------+------------------------------- a00baafb-1d3e-49b0-b3e9-4aa21df87716 | 2024-11-08 19:16:08.869313+00 a00baafb-1d3e-49b0-b3e9-4aa21df87716 |

Finally i started the server again with the workers, and re-ran the query:

content=# select template_uuid, deleted_at from templates_repository_configurations ; template_uuid | deleted_at --------------------------------------+------------ a00baafb-1d3e-49b0-b3e9-4aa21df87716 |

so i think its good!

jlsherrill · 2024-11-08T14:33:00Z

[test]

jlsherrill · 2024-11-08T15:14:55Z

[test]

jlsherrill · 2024-11-08T19:13:30Z

[test]

jlsherrill · 2024-11-08T19:31:16Z

i did discover that i wasn't updating repository_path in the snapshots table, so i added that as part of 1e5b426 and added an integration test

jlsherrill · 2024-11-11T01:04:55Z

[test]

xbhouse · 2024-11-11T16:44:49Z

repository paths look good now!

after running rename-domains, i'm still seeing the delete-repository-snapshots task fail on deleteRpmDistribution when deleting a repo created previously though:

"error reading task: 404 Not Found: \n<!doctype html>\n<html lang=\"en\">\n<head>\n  <title>Not Found</title>\n</head>\n<body>\n  <h1>Not Found</h1><p>The requested resource was not found on this server.</p>\n</body>\n</html>\n"

also when fetching a snapshot task in the UI for existing repos (which uses /admin/tasks/:uuid):

"error: code=500, title=Error fetching task, detail=error: code=500, title=Error parsing task payload, detail=error reading task: 404 Not Found: \n<!doctype html>\n<html lang=\"en\">\n<head>\n  <title>Not Found</title>\n</head>\n<body>\n  <h1>Not Found</h1><p>The requested resource was not found on this server.</p>\n</body>\n</html>\n \n \n"

i think the pulp hrefs in our db need to be updated to the new domain too? i see the hrefs using the new domain in pulp:

/api/pulp/cs-65f27c79/api/v3/distributions/rpm/rpm/01931c0c-999b-7c2b-9654-cbd2719b949b/
/api/pulp/cs-65f27c79/api/v3/repositories/rpm/rpm/01931c0c-8a47-73b0-bab7-cc2f05e239e7/versions/1/
/api/pulp/cs-65f27c79/api/v3/publications/rpm/rpm/01931c0c-9518-77d1-a5d5-a308bcca9579/

but not in our db:

/api/pulp/65f27c79/api/v3/distributions/rpm/rpm/01931c0c-999b-7c2b-9654-cbd2719b949b/
/api/pulp/65f27c79/api/v3/repositories/rpm/rpm/01931c0c-8a47-73b0-bab7-cc2f05e239e7/versions/1/
/api/pulp/65f27c79/api/v3/publications/rpm/rpm/01931c0c-9518-77d1-a5d5-a308bcca9579/

swadeley · 2024-11-13T09:47:46Z

Hi @jlsherrill , please rebase.

jlsherrill added the no-qe-needed label Oct 25, 2024

jlsherrill force-pushed the 4822_1 branch 5 times, most recently from c2a331f to c2834dd Compare October 31, 2024 20:30

jlsherrill force-pushed the 4822_1 branch 2 times, most recently from f3607cb to 5c4e1df Compare November 1, 2024 19:52

jlsherrill marked this pull request as ready for review November 1, 2024 20:01

jlsherrill force-pushed the 4822_1 branch from 2774920 to fe09e98 Compare November 4, 2024 21:59

xbhouse self-assigned this Nov 5, 2024

xbhouse reviewed Nov 5, 2024

View reviewed changes

pkg/jobs/rename_domains.go Outdated Show resolved Hide resolved

jlsherrill mentioned this pull request Nov 5, 2024

Fixes 4952: add snaps list with temp list/fetch #874

Open

rverdile reviewed Nov 6, 2024

View reviewed changes

jlsherrill force-pushed the 4822_1 branch from d9cbae8 to bad326c Compare November 18, 2024 19:17

jlsherrill added 2 commits November 19, 2024 11:12

Fixes 4822: rename domains with cs- prefix

e5e065b

Changeup relationship for template_repository_configurations

8f4c4df

jlsherrill added 7 commits November 19, 2024 11:12

fix bug where env prefix was using wrong domain

efeb78b

use correct domain name for lookup

9e20482

adding rename domains test

35babb6

Add integration test and rename snapshot repo paths

b728bd8

Fix doman spelling

47c320d

backup data and rename hrefs

23be27f

handle dist hrefs

61a7116

jlsherrill force-pushed the 4822_1 branch from c0f28ff to 61a7116 Compare November 19, 2024 16:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes 4822: rename domains with cs- prefix #865

Fixes 4822: rename domains with cs- prefix #865

jlsherrill commented Oct 25, 2024 •

edited

Loading

jlsherrill commented Oct 25, 2024

xbhouse commented Nov 5, 2024

xbhouse Nov 5, 2024

xbhouse Nov 5, 2024

jlsherrill Nov 5, 2024

xbhouse commented Nov 5, 2024

jlsherrill commented Nov 5, 2024

xbhouse commented Nov 5, 2024 •

edited

Loading

rverdile Nov 6, 2024

jlsherrill Nov 7, 2024

rverdile Nov 7, 2024

jlsherrill Nov 8, 2024

jlsherrill commented Nov 8, 2024

jlsherrill commented Nov 8, 2024

jlsherrill commented Nov 8, 2024

jlsherrill commented Nov 8, 2024

jlsherrill commented Nov 11, 2024

xbhouse commented Nov 11, 2024 •

edited

Loading

swadeley commented Nov 13, 2024

Fixes 4822: rename domains with cs- prefix #865

Are you sure you want to change the base?

Fixes 4822: rename domains with cs- prefix #865

Conversation

jlsherrill commented Oct 25, 2024 • edited Loading

Summary

Testing steps

jlsherrill commented Oct 25, 2024

xbhouse commented Nov 5, 2024

xbhouse Nov 5, 2024

Choose a reason for hiding this comment

xbhouse Nov 5, 2024

Choose a reason for hiding this comment

jlsherrill Nov 5, 2024

Choose a reason for hiding this comment

xbhouse commented Nov 5, 2024

jlsherrill commented Nov 5, 2024

xbhouse commented Nov 5, 2024 • edited Loading

rverdile Nov 6, 2024

Choose a reason for hiding this comment

jlsherrill Nov 7, 2024

Choose a reason for hiding this comment

rverdile Nov 7, 2024

Choose a reason for hiding this comment

jlsherrill Nov 8, 2024

Choose a reason for hiding this comment

jlsherrill commented Nov 8, 2024

jlsherrill commented Nov 8, 2024

jlsherrill commented Nov 8, 2024

jlsherrill commented Nov 8, 2024

jlsherrill commented Nov 11, 2024

xbhouse commented Nov 11, 2024 • edited Loading

swadeley commented Nov 13, 2024

jlsherrill commented Oct 25, 2024 •

edited

Loading

xbhouse commented Nov 5, 2024 •

edited

Loading

xbhouse commented Nov 11, 2024 •

edited

Loading