Replies: 17 comments 39 replies
-
|
@mshanemc this looks amazing. Might it make sense to standardize on a single token syntax? ie: Also, would you consider shell completion ? Imagine {
// glob version
"file": "*.namedCredential-meta.xml",
"toReplace": "@@username@@,"
"replaceWithEnv": "$(my shell command)"
}This would effectively enable external vault support with no-per-vault cost. This might only work for people on sane shells. |
Beta Was this translation helpful? Give feedback.
-
|
Shell expansion--I can see the downside for teams where users don't share OS/shell behavior. |
Beta Was this translation helpful? Give feedback.
-
|
My biggest worry on choosing a token syntax is that some new feature decides to use that syntax in its metadata. The practical downside is that it would preclude example 3, where you want a valid value that goes to most environments that can conditionally be replaced for other environments (ex: prod). |
Beta Was this translation helpful? Give feedback.
-
|
OK, this is decided. See |
Beta Was this translation helpful? Give feedback.
-
|
@mshanemc - while not perfect, I have verified that WSL2 enables bash, zsh, fish and others. Meaning Shell expansion is available on all platforms - so long as you choose the appropriate shell. Shell expansion would be amazing here. |
Beta Was this translation helpful? Give feedback.
-
|
Since most of the metadata is XML, it could be useful to support XPath as well. This would allow to target specific fields easily, based on the metadata structure. Targeting a specific field with Regexp can be very tricky if not impossible. |
Beta Was this translation helpful? Give feedback.
-
|
@mshanemc Excited to see this is coming to life!
|
Beta Was this translation helpful? Give feedback.
-
|
The scan would always be limited to the files being deployed. I'd like a full deploy (ex: pushing to a scratch org) to have the same behavior as the packaging operation. Those large deploys are where I'm more concerned about perf than the smaller ones. Having the filenames explicit would help prevent unintended replacements (typo where you accidentally used I think if you want full project scans, you could just provide a very greedy glob (**/*) like example 2? Would that work? |
Beta Was this translation helpful? Give feedback.
-
|
I'm really not sure how to do your tree design where it's always the same file, but relative to a different path based on envs without it getting out of hand. |
Beta Was this translation helpful? Give feedback.
-
|
@imTachu @mshanemc the suggestion of using something like |
Beta Was this translation helpful? Give feedback.
-
|
@mshanemc I had that for a while in my sfdx plugin DXB. sfdx dxb:org:data --config config/data-env-def.json --environment SIT This came quite early when sfdx plugin was made available and for 2 main reasons: scratch org creation was a bit of a pain back in the days ( can we say that now?) and also because many of my clients have metadata specific requirement. What you suggested is pretty much what I did, exact file path, the regex and what value to replace it with. My dxb command just require you to pass the json definition file and I would run the command before a deployment. You could have 2 approaches : one file per env or one single file for all environment but in that case you have to pass the alias of the environment which was used as a key property. Example dxb def file: {
"ST": [
{
"pagePath": "force-app/main/default/workflow/Account.workflow-meta.xml",
"replacerules" : [
{
"source": "{{username}}",
"target": "[email protected]"
}
]
}
],
"SIT": [
{
"pagePath": "force-app/main/default/workflow/Account.workflow-meta.xml",
"replacerules" : [
{
"source": "{{username}}",
"target": "[email protected]"
}
]
}
]
}Quick example of use in azure pipeline: - script: |
if [[-s "$(sfdx-env-mapping)"]]
then
sfdx dxb:org:data --config $(sfdx-env-mapping) --environment "${{ parameters.TARGETENV }}"
fi
displayName: "Set specific org data"I like the idea to target multiple files using the wildcard maybe I should improve my command 🤣 or just wait for you to do it. |
Beta Was this translation helpful? Give feedback.
-
|
@imTachu Did you consider using a templating solution such as Nunjucks for your use case ? It's much more flexible, because metadata templates can include conditions, formulas, variables etc. And, it's very quick. |
Beta Was this translation helpful? Give feedback.
-
|
It looks like https://github.com/jayree created something similar, which is flexible and functional from my perspective, (see sfdx jayree:org:configure command here https://github.com/jayree/sfdx-jayree-plugin and configuration example here: https://github.com/jayree/sfdx-jayree-plugin/blob/v4.4.3/.sfdx-jayree-example.json) |
Beta Was this translation helpful? Give feedback.
-
|
@mshanemc this is a great feature, currently we rely on deployment scripts and
|
Beta Was this translation helpful? Give feedback.
-
|
@julianduque - Maybe I mis understood something here, but @mshanemc I really want to encourage you to build this in such a way that the metadata replacement is ephemeral. Ie: making a push or deploy should result in no-new-change to the existing codebase. Rather it should modify a deploy-only copy of the metadata. When the deploy(or push, simply saying deploy from now on) completes, that version is eliminated from disk - if it has to be on disk. Ideally, the modified files would be in memory only. |
Beta Was this translation helpful? Give feedback.
-
|
@codefriar yes, that's a better approach. I'm comparing to my current approach in |
Beta Was this translation helpful? Give feedback.
-
|
we do not want to put anything on disks. these "changes" shouldn't show up in git (which might be committed), or in local source tracking, or your computer's tmpDir or trash folders given that secrets is one of the use cases. |
Beta Was this translation helpful? Give feedback.
-
|
@mshanemc fantastic then! |
Beta Was this translation helpful? Give feedback.
-
|
Regarding the behavior for Retrieve: Spurious changes to metadata because of a retrieve are a perennial source of frustration -- depending on the care/ability of the engineer, things often get committed to source that should've been reverted because they just don't know better or the breadth of the diff that came down from the server makes it hard to spot the one line that shouldn't have been changed. (i.e. if you pull down a Perm Set, there might be dozens of lines changed because of a new field/object that were added .. it would be easy to overlook that the placeholder was overwritten with a concrete value when doing code review) I really think that the extra work to effectively "reverse" the merge is necessary. I'd expect that when the new metadata comes down, it's compared with the local file and any of the placeholder values are replaced in the source file before it's written to disk. |
Beta Was this translation helpful? Give feedback.
-
|
Call me a cynic, I like the idea of this feature a lot; however; I don't want the SDR library to build/own it directly. I want the SDR library to provide the hooks so that another tool can be built. I'm totally fine with the CLI tools team building the first pass at that "modify source in flight" tool, and aside from some edge cases the designs look pretty good. I would really like others to be able to come along and do something awesome with the same hooks. Baking it into the SDR feels like crossing responsibilities somewhat and limiting what the community could do here. We have some metadata that we keep "scratch org ready" in source control but we run a pre-deploy process to manipulate that metadata when deploying to production, those manipulations are not committed. We have some processes that do that, a simple config file that understands how to blow out into the necessary multi-node XML changes ( I don't store the full XML replacement text, only the variable portions). I would love to change that process to only impact in-flight records, but the proposed solution, while possible, would be more cumbersome to manage because I'd possibly have a lot of per-rule replacements. I could see that section of the If SDR created a hooks-like extension point I don't know how we "register" extensions, probably another node in the If you've already evaluated and shot down this idea, I'd love to understand that decision. |
Beta Was this translation helpful? Give feedback.
-
|
The CLI is hooks-friendly, and used to support this kind of thing (see blog). That model would work fine if everything was always done via the CLI. This design targets SDR because the Salesforce VSCode Extensions' deploy action doesn't use the CLI. The explanation behind that: they got faster perf, especially on small operations, by not having the CLI do all the stuff it usually does (loading help, flag validation, etc) and getting to calling SDR to deploy stuff sooner. Without the CLI, custom plugins' hooks aren't called. It's potentially possible to rearchitect how the vscode extensions work to register hooks as a more flexible alternative, but not something CLI can do alone. Aside from that, SDR works in a highly optimized, pipes/streams way. We still emit hooks (blog, again), but I'm not sure we could make hooks works as you describe. |
Beta Was this translation helpful? Give feedback.
-
|
I'm thinking about just having a plugin system for SDR. Everything else around that is details. Plugins would have lifecycle injection points that gives the name of the file, the type of metadata, and the contents en-route to the deployment. They can manipulate those contents as part of the stream. I'm thinking something closer to middleware in an HTTPS server. Registering those plugins would need a home. Depending on how SDR is setup, it would be nice to load them only as the library is initialized, not have to load them from disk for every action. Would it be helpful if I mocked something on a fork of the library? Or is the book closed on this one and I should move on? (No hurt feeling if this just isn't the direction) |
Beta Was this translation helpful? Give feedback.
-
|
I want a configuration-only option for replacements, and plan to work on that next week, but I'd be very interested in a DIY-hooks model for the "everything else" and appreciate the help. |
Beta Was this translation helpful? Give feedback.
-
|
I really like this idea @mshanemc, thanks for bringing this to everyone's attention! Issues With the Status QuoThe lack of environment variable support for Salesforce projects has led to a terrible consequence within the ecosystem; the proliferation of multiple long-lived branches which correspond to sandboxes. This multi-branch strategy makes the act of a Generally speaking I think there are a few types of metadata we want to treat differently:
The value stored in these types often need to support different values based on where the sfdx project is being deployed to. These values are used in a whole bunch of metadata types throughout the entire platform:
What We've Done So FarMy team has tried to solve part of this and we shared our findings at Dreamforce 2022 - here is our presentation, pages 26 and 27 are of the most relevance. What we've done is define a protocol buffer which has the schema definition for an "Environment" object. The environment proto has values such as an array of environments/qa.environment.textproto This naming scheme and folder structure enable us to identify all of the environment specific string metadata overrides from a single environment variable injected into our build tool; When defining the object model for the schema replacement, the structure must support all necessary identifiers to find the metadata you are trying to replace and the replacement value for the metadata. Custom Label is an easy example with Within the runtime of our build tool's VM, we execute a command line script which roughly looks something like this: some/path/to/some/executable setSalesforceEnvironmentVariables --environment=$TARGET_SALESFORCE_ENVIRONMENTThis will read the environment value, use it to identify the <labels>
<fullName>America_Service_Email</fullName>
<language>en_US</language>
<protected>false</protected>
<shortDescription>America Service Email Address</shortDescription>
<value>[email protected]</value>
</labels>
<!–--XML is updated right before deployment to QA environment–-->
<labels>
<fullName>America_Service_Email</fullName>
<language>en_US</language>
<protected>false</protected>
<shortDescription>America Service Email Address</shortDescription>
<value>[email protected]</value>
</labels>This mechanism allows for the safe deployment of a single branch to multiple target Salesforce environments. It doesn't introduce much noise or overhead within the repository itself, as these node value swaps are made in our build tool's ephemeral runtime, then expire. Regarding Placeholders - ex:
|
Beta Was this translation helpful? Give feedback.
-
|
thanks for the insights!
I definitely want to preserve that option. I'm going to modify the proposal to explain the rationale for keeping it user-defined. See |
Beta Was this translation helpful? Give feedback.
-
|
How are variables loaded? Behaviors ...and would deploy fine in the "warning" scenario, but I don't want it to deploy. I want it...need it... to fail. These packaging scrips often run in Github Actions, and there isn't a human monitoring for warnings. An error would/should stop the deployment and packaging script from continuing. Challenges - Retrieve |
Beta Was this translation helpful? Give feedback.
-
|
It will inject secrets on deployment but won't modify the files on the field system. Pulling server-side change would then pull down and overwrite files, potentially with secret values, but this is an existing problem today with all toolsets. I just don't think it is this tool's responsibility to control what does or does not get committed in version control. Other options and tools are available today to prevent secrets from being committed. I agree it'd be cool to maintain the placeholders on the retrieve, but I'd prefer to have something rather than nothing. I can address the accidentally committing secrets issue myself with husky...I can't address this secret inject issue without SDR enhancements. |
Beta Was this translation helpful? Give feedback.
-
|
@tehnrd returning to the Would the dotenv cli be sufficient to let you read some .env file(s) so that SDR's responsibility is to check the env and you're free to set them however? ex: |
Beta Was this translation helpful? Give feedback.
-
|
@mshanemc ...but how would that work if I am in VSCode, save a file, and the push command automatically runs? I don't have control of this. I really don't want to create a custom command to save/deploy, I want to save the file in VSCode (CMD+S) and it "just works." This is how my plugin worked before it broke and was a great UX. Admittedly, without something like dotenv I probably won't use this feature. I have a lot of different projects on my machine, and they all have different environment variables, some with the same name, so setting these in some global .zsh config file isn't an option either as the env vars are project specific. I understand not want to add a dependency, but dotenv is one of the most popular and well-maintained packages. It is almost becoming the industry standard to use in JS/TS projects. If you want to avoid a dependency, you could also just replicate the basic functionality to read from .env files in the project as it is not a terribly complex package/code. |
Beta Was this translation helpful? Give feedback.
-
|
Using And as @tehnrd said, using |
Beta Was this translation helpful? Give feedback.
-
|
I forgot to mention that |
Beta Was this translation helpful? Give feedback.
-
We could use a cache file to allow the discovering of files with placeholders without having to explicitly configure the file in PMD uses a similar mechanism to run the analysis only on changed files. |
Beta Was this translation helpful? Give feedback.
-
|
interesting. We'd also have to deal with the other scenario where replacement config changes (new/modified placeholders, etc) ...just toss out the cache? |
Beta Was this translation helpful? Give feedback.
-
|
We shipped this (today's release where I didn't add |
Beta Was this translation helpful? Give feedback.
-
|
@mshanemc is this live? I'm on 7.181.1 (Dec 15, 2022) [stable] and when I create a |
Beta Was this translation helpful? Give feedback.
-
|
You can replace values from environment variables or text in a file. Not sure why'd you want to do a straight-up string replacement like that. The feature is primary for secrets and other environment-level configs. |
Beta Was this translation helpful? Give feedback.
-
|
No reason, I was just playing with it. |
Beta Was this translation helpful? Give feedback.
-
|
Cool cool, saw adding |
Beta Was this translation helpful? Give feedback.
-
|
It would be interesting to be able to replace (conditionnally) by a straight string (or current username) when deployibg EmailFolder and EmailTemplateFolder to scratch org, to change shares or org-wide email addresses in queues etc. |
Beta Was this translation helpful? Give feedback.
-
@RupertBarrow I have recently developed a plugin with a predeploy hook that could be interesting for you: Please let me know if this works for you or if you've found another solution in the meantime. |
Beta Was this translation helpful? Give feedback.
-
|
@mshanemc This doesn't appear to work on file saves or source pushes in VSCode. Is this expected? I thought this being a first-party feature built into the core CLI would cause it to be run everywhere. I created an issue here forcedotcom/salesforcedx-vscode#4629 |
Beta Was this translation helpful? Give feedback.
-
|
@mshanemc Is it possible to utilize this in unlocked packaging ? or any future plans for unlocked packaging. (Could be a different team entirely) Example. This feature sounds like it would be ideal, but wasnt sure how to work it, if at all with unlocked packaging. |
Beta Was this translation helpful? Give feedback.
-
|
you can set the env If you're using sensitive info in replacements, set the env on the command itself (don't export and leave it in the env) to avoid accidentally writing secrets to the filesytem. |
Beta Was this translation helpful? Give feedback.
-
|
closing this since we've gotten the MVP shipped (and windows path stuff resolved). Please open any feature requests as new discussions! |
Beta Was this translation helpful? Give feedback.
-
|
@mshanemc @tehnrd - Do string replacement run prior to packaging a 2gp managed package? The docs only mention about deploying to an org and I have a custom solution I built as a node script that runs before an after my packaging process, but would love to throw that out if this works for packaging of 2gp managed packages. (I have some protected custom metadata where I don't want secrets in the repo). |
Beta Was this translation helpful? Give feedback.
-
|
Yes, see #1755 (reply in thread) |
Beta Was this translation helpful? Give feedback.
-
|
@shetzel - I have not been able to get this to work with packaging When deploying to a scratch org, the command errors out based on the replacements configuration - but the same such error does not happen during packaging, which tells me that the replacements are not being considered. I am using a protected custom metadata component, so I cannot see the value in the target org to 100% confirm - but authentication is broken, so I am sure the replacement did not get included in the deployment. EDIT: Got it working, I had to set |
Beta Was this translation helpful? Give feedback.
-
|
@paustint - you have good timing! As you discovered, the way to get this supported is to set In fact, if you want to test it out just install the latest packaging plugin explicitly and you'll get that change. You should probably uninstall it after testing though so you stick with the packaging plugin built with the CLI version (uninstalling takes you back to the "core" version). |
Beta Was this translation helpful? Give feedback.
-
Let's discuss a new feature I'll be working on soon.
Background
Jason Venable asking for this on twitter https://twitter.com/TehNrd/status/1552655461790257158
hooks blog: https://developer.salesforce.com/blogs/2022/02/we-broke-deploy-retrieve-hooks
Credit also to the sfdx folks at Picnic for some of the ideas/requirements.
Idea: replace placeholder strings in metadata during a deployment
Primary use cases
Location
This feature should go into SDR so that it's invoked from sfdx's and sf's deploy/push commands, from vscode, and from the new packaging commands (which use SDR).
An sfdx plugin alone would not allow Salesforce's vscode extensions to inherit the behavior.
Use via SDR would allow custom plugins using the SDR library to inherit the behavior, too.
Constraints
For perf reasons, we do NOT want to read every file looking for strings. So we want you to indicate by filename or glob what the target files are.
Configuration
Adds a new
replacementsproperty tosfdx-project.jsonEx 1: replace a string in a file with a value from the env
Ex 1b: regex
Ex 2: target multiple files via a glob.
Sometimes its not a secret, so you can store them in separate file the you could check in.
Ex 3: conditionally replacing from a file based on one or more envs matching. Here, the test-endpoint gets replaced with the country-appropriate prod endpoint during a deployment. Imagine one of these for each country
Placeholder format
Rather than dictating a format (ex:
{{ foo }}) we will leave it up to the users. You can use a string or regex. Why?Behaviors
No match for a file
File matches, but no match for the toReplace string
fooinbar-meta.xmlbut it’s not there. Check your sfdx-project.json for wrong or unnecessary replacementsFile matches, toReplace string is found, but the env doesn’t exist
❓ would you want the filenames/placeholders to show up in the deploy/push output so you can confirm that what you expected to happen did?
Challenges
Retrieves
pullthe sourceIn either case, the metadata retrieve would not contain your placeholders, but whatever concretized value you were using placeholders to avoid.
❓ We could give you a warning that your placeholder just got overwritten? Is that enough?
💡 We could try to be really clever and check the replacements on the way down (basically doing it backwards since you've supplied a list of files and envs). I can see this getting confusing if you've changed the environment values or are working on a different org.
Out of scope for now
Changes
toReplaceto replaceString` and replaceRegex and added a regex examplePlaceholder formatsection to communicate decisionBeta Was this translation helpful? Give feedback.
All reactions