Fire preretrieve hook even when metadata is deleted (or add another hook)

[FabienTaillon]

Is your feature request related to a problem? Please describe.
When a metadata is deleted from the scratch org, technically nothing is retrieved via a package.xml with sfdx force:source:pull, and the preretrieve hook isn't fired.

What are you trying to do
There are currently some issues with the pull command (mostly because of dependencies not being tracked on the org in SourceMember). For instance, when you delete a field that is referenced in a Permission Set, the field is deleted locally but the related Permission Set isn't pulled. I would like to be able to add metadata to retrieve in this hook, even if no metadata is supposed to be retrieved (but something is "pulled", like a deletion).

Describe the solution you'd like
See the preretrieve hook firing on pull even if there are just deletions, and thus no package to retrieve so far. Also, having an additional attribute in the result type (similar to what we get with force:source:status with the changed metadata and state) would be a plus.

Describe alternatives you've considered
I tried using the postretrieve hooks which works, but need to retrieve a second time, plus this break the source tracking (I guess there is a way to make this work).

[github-actions[bot]]

Thank you for filing this feature request. We appreciate your feedback and will review the feature at our next grooming or sprint planning session. We prioritize feature requests with more upvotes and comments.

[uip-robot-zz]

This issue has been linked to a new work item: W-8972481

[FabienTaillon]

I've tried this with the new source tracking commands beta #1258, and it's even more confusing.

Previously with force:source:pull:

PreRetrieve wasn't firing, but PostRetrieve and PostSourceUpdate were. This wasn't really coherent as nothing was technically retrieved (just a field was deleted locally). So either PreRetrieve and PostRetrieve should both fire, or none of them should.

With the new commands, it's actually more coherent:

Nothing is fired. If I understand correctly, PostSourceUpdate will be removed, and as nothing is really retrieved, other hooks are not firing.
The issue here is that in this specific case (only some metadata deleted from the org), we don't have any hooks that we can use anymore.
To me, we don't care that the metadata doesn't retrieve anything technically speaking, we're actually pulling something, a deletion change, so both PreRetrieve and PostRetrieve should fire.

What do you think @mshanemc @shetzel ?

[FabienTaillon]

Looks like it's done on purpose here.

The use case of this hook, as per the documentation, is "Retrieve additional metadata or configuration when a specific source file is retrieved". I don't see why I would want to retrieve additional metadata when a file is retrieved but not when a file is deleted.

Also, with the previous hook for force:source:pull, we had access to the packageXmlPath, so I was able to edit the file to add the additional metadata I wanted to retrieve, and everything was retrieved in one row by the standard command.

With the new hook giving us a list of MetadataComponent, how I am supposed to retrieve additional components ? Manually do it, with another retrieve than the one performed by the command ?

[mshanemc]

Thanks for the feedback and for testing the new beta commands. Most of the changes relate to the new SDR library, which is way, way more performant than the old code (times shown are ms). Push/pull (beta) are using the same library.

One of the main ways it achieved that is by reduces FS operations (this is particularly a slowness cause for windows)...it generates the zip file that goes to the metadata API in memory, streaming all the converted components. So there is no packageXmlPath because we never saved a package.xml locally (except if you set SFDX_MDAPI_TEMP_DIR because you wanted to keep those artifacts on purpose). And there's really not a good way for users to intercept that flow of metadata through the streams--some files might be already in the in-memory zip before the entire deploy dir is even finished being read.

One option to get what you are requesting would be to fire another preRetrieve hook from inside here with the deletes.

The real problem (if I understand what you're doing with hooks) is that the old hooks let you access the "in flight" metadata and make changes to it before the deploy/retrieve, where the new ones are more "here's what's going on" but you can't change the deployment at that time because it doesn't exist locally. So you would need to do your own separate transaction outside of the one the CLI is doing.

I know that's going to be bad news.