Service broker can declare supported schema for parameters field

[bmelville]

Service broker will provide schema information for both service instances and bindings for each plan. The schema information is for both parameter inputs and output properties. Currently the proposal is that the schema format will be JSON schema v4.

Proposal: https://docs.google.com/document/d/1-IKI-PwPnhtbK0su1UzWA_UoyaCjL8S2d6ky3Xmuwvg/edit?usp=sharing

[shalako]

This issue seems more appropriate for the Kubernetes service-catalog project. The openservicebrokerapi org is not meant for development of platform controllers.

[bmelville]

@shalako the intention of this issue is for the parameter schema PR. I've renamed to make it more appropriate.

[bmelville]

Proposal doc: https://docs.google.com/document/d/1-IKI-PwPnhtbK0su1UzWA_UoyaCjL8S2d6ky3Xmuwvg/edit?usp=sharing

[shalako]

@avade I know you're planning to draft a proposal for the CF community for this feature. You may want to consider Brendan's existing google doc; link to it and dedup or duplicate the content in yours? I expect going forward we'll have to have a design proposal for the API and another for CF changes.

[avade]

I had already lifted much of the content form Brendan's doc :) Brendan are you going to update the proposed API spec part of this doc or sharing more as an example? If this is going to be "the" proosal doc for this feature, can I get rights to suggest edits, please?

[bmelville]

Hey Alex, The API spec part was more an example section, but I'm happy to make this more the canonical doc if you'd like. I believe that you can suggest edits if you have comment rights by switching to suggest mode. If not I can give you edit rights. B.

…

On Tue, Nov 29, 2016 at 11:40 AM, Alex Ley ***@***.***> wrote: I had already lifted much of the content form Brendan's doc :) Brendan are you going to update the proposed API spec part of this doc or sharing more as an example? If this is going to be "the" proosal doc for this feature, can I get rights to suggest edits, please? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#59 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKwU8r_gzpQAj5w7o34U5k1hEWU42AgOks5rDH-fgaJpZM4JqRle> .

[avade]

I have had a read through the comments and found it difficult to work out the state of the discussion.

Perhaps @bmelville / @gberche-orange could provide a summary and what needs to happen before we can accept the proposal and get started on a Cloud Foundry implementation.

[gberche-orange]

To my understanding, the core specification needs a decision on where schemas should be attached at :

• object level (service instance, service binding) to the expense of restricting some fields only to some verbs (create/update/delete/actions)
• method level (provision, bind, update, delete actions) to the expense of duplicating field definitions for the ones common between verbs, such as create/update.

The JSON schema extensions (x-serviceBroker-updatable, x-servicebroker-supported-in-verbs, or x-servicebroker-confidential, x-servicebroker-displayType, x-servicebroker-service-instance-required-types, x-servicebroker-service-instance-required-tags ) seem less mature and needs more explorations of the use-cases to support.

The core specifications (a.k.a. without schema extension) could a good candidate for a 2.12 feature with an experimental status. This would enable to start getting feedback from prototype implementations in CF such as:

• CF web UIs
• the CF CLI (distributed as a plugin) that would perform
  • arbitrary params validation
  • arbitrary params prompting (even if this is not currently the direction adopted by the CLI team /CC @dkoper)
  • arbitrary params completions
• generic service brokers (similar to pivotal's on-demand-service broker) leveraging the schema to validate arbitrary params
• CC validating the schema (although that's not mandated in the specs)

Based on feedbacks from such prototypes, decisions could be taken on

• whether to remove the experimental status on core schema specs or amend them
• which schema extensions to add to fullfill the missing use-cases

[bmelville]

I think that's a great overview from Guillaume. The remaining work item is the decision on where to hang schemas, and there will likely be more to flush out, especially around the new Broker Actions proposal. I think we have landed on method-level schemas, but it seems like there is definitely room for a lot of exploration and understanding of use-cases.

…

On Thu, Dec 15, 2016 at 1:30 PM, Guillaume Berche ***@***.***> wrote: To my understanding, the core specification needs a decision on where schemas should be attached at : - object level (service instance, service binding) to the expense of restricting some fields only to some verbs: create/update/delete/actions) - method level (provision, bind, update, delete, actions) to the expense of duplicating field definitions for the ones common between verbs, such as create and update The JSON schema extensions (x-serviceBroker-updatable, x-servicebroker-supported-in-verbs, or x-servicebroker-confidential, x-servicebroker-displayType, x-servicebroker-service- instance-required-types, x-servicebroker-service-instance-required-tags ) seem less mature and needs more explorations of the use-cases to support. The core specifications (a.k.a. without schema extension) could a good candidate for a 2.12 feature with an experimental status. This would enable to start getting feedback from prototype implementations in CF such as: - the CF CLI (distributed as a plugin) that would perform - arbitrary params validation - arbitrary params prompting (even if this is not currently the direction adopted by the CLI team /CC @dkoper <https://github.com/dkoper>) - arbitrary params completions - generic service brokers (similar to pivotal's on-demand-broker) leveraging the schema to validate arbitrary params - CC validating the schema Based on feedbacks from such prototype, decisions could be taken on - whether to remove the experimental status on core schema specs or amend them - which schema extensions to add to fullfill the missing use-cases — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#59 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKwU8qnDJGQQq3Orkr_Cn82QpyPd6jX9ks5rIbFsgaJpZM4JqRle> .

[bmelville]

Hey folks,

I have updated the proposal with the method-based schema approach, and have also added a section on extensions (outside the scope of this but worth discussing a bit since there have been so many questions).

Please take another look when you have a chance.

B.

[shalako]

I've read the new proposal. Looks good.

[gberche-orange]

The "plan-attached schemasets" concepts were reviewed during the Feb 02, 2017 design session. The proposal has been updated with details, comparison table updated, along with other proposal archived for reference.

Feedback is welcome.

Next steps one feedback is collected would be:

• Validate this proposal by implementation in CF platform
• update the dependent proposals (e.g. Generic Extensions (Actions) Proposal #114, Ability to declare schema of binding credential output #116 and possibly Remove some CF-isms #115) to validate extensibility support.

[mattmcneeney]

Hi all,

As discussed in the calls earlier this week, we've made some provisional spec changes for two of the proposals:

1. Proposal 5: Service-level schema-sets
   Proposal document
   Provisional spec

2. Proposal 2: Method-attached schemas
   Proposal document
   Provisional spec

Feedback welcome. We discussed creating spike implementations for both of the above proposals in Cloud Foundry as a good next step.

[petereberlein]

Personally I would have preferred to keep the actual schemas out of the catalog, either using a dedicated endpoint or referencing URLs for each schema, but if the majority of the group prefers an all-in-one approach then I would vote for option #2, method attached schemas. Although they increase redundancy at least it is obvious which schema applies when, which contributes to simplicity. Also, if in future different osb platforms are reflected by different catalog responses (requested with an osb platform specific query), then the need for extending the more generic applies-when clause from proposal #1 is less relevant.

[shalako]

I spoke with the PM for the Cloud Controller, which is the client of the broker in CF. He can work with either proposal. We discussed they're likely to story the schemas with plans, even if the broker exposes them at the service level. So if a schema has no applies_when.service_plan_id, then Cloud Controller will save the schema for all plans of the service. CF clients would then get schemas only for requested plans, and either parse them by applies_when or by nested object/method.

We'll send these proposals to our CLI and web client teams for feedback.

[gberche-orange]

Also, if in future different osb platforms are reflected by different catalog responses (requested with an osb platform specific query), then the need for extending the more generic applies-when clause from proposal #1 is less relevant.

@petereberlein @shalako

Rethinking about last WG call discussion on having the catalog endpoint request convey the platform profile (e.g. in a context parameter): while this enables the broker response to restrict the schemas returned to the only requested platform (say CF), this is likely to make the broker federation use-cases more complex:

In the case of a federation of brokers (e.g. where a federating broker federates a number of federated brokers, supporting different platforms and possibly different OSB API versions), the federating broker likely needs to obtain all supported schemas by a federated broker at once, and would prefer avoiding to repetitively query the /catalog endpoint with all possible platforms in a context parameter or all possible X-Broker-API-Version header variations.

I still believe keeping room for future extensibility (without introducing breaking changes, such as with the applies-when construct) is useful and relevant, as we explore new use-cases and broader OSB API usage.

I hope that the "validation by implementation" phase in CF will provide additional useful feedbacks for both proposals #5 and #2.

Sorry I won't be able to make it to the weekly call today.

[shalako]

@gberche-orange I expect in that scenario, the federating broker would have to fetch all the schemas. A query param to filter the response would be an optional nice-to-have.

I have spoken with the team responsible for the Pivotal web UI for CF. They have expressed a preference for the simpler, more explicit declaration of schemas enabled by proposal 2.

[mattmcneeney]

I think that proposal 2 (method-level schemas) will be easier for platform providers to implement (they don't have to match schemas to plans, methods and operations) and easier for service broker authors to implement. Both proposals have capacity for future extensibility, so my preference would be to proceed with a test implementation of proposal 2.

[mattmcneeney]

We probably won't be able to discuss this on tonight's call since we're just discussing 2.12 features, so if we could get some feedback on this issue here, that'd be super helpful for us. Thank you!

[skaegi]

My team (Bluemix Devops Services) has a controller that manages dev tools vs services but is otherwise very similar to CF and borrows many concepts like provisioning, binding, etc. We've been using JSON Schema for validation of parameters as well as a JSON Schema Form variant for UI for a few years now and by and large it has worked well for us.

Based on our experience we're in the camp that says JSON Schema is a good choice for input parameter validation. I think there is still need for broker-based custom validators but I believe that can be built as an extension of JSON Schema (and written up in a separate issue).

JSON Schema is relatively stable and there is an active community pushing a new round of standardization. For a UI binding for JSON Schema there is still a lot of thinking going on in the Schema Form space. It is coalescing with efforts like JSON Schema Form however still are a number of libraries that make subtly different choices.

I would suggest that JSON Schema validation is mature enough and should be considered for addition. I think it would make sense to open a separate issue for the UI binding and track it independently. I'd also be very hesitant to add any new keywords like applies-when and instead be verbose and use JSON references to avoid repeating definitions.

I would also not add any new UI specific keywords to the schema at this point until we're sure this is a good idea. In our experience mixing UI and Schema is a bit of an anti-pattern and have found the structure of the schema and form are the same only simple cases where sections and choice are not involved.

What we ended up doing was adding one keyword form where we put all of our UI info but otherwise using JSON Schema as-is. Here's a simple example:

"parameters": {
    "$schema": "http://json-schema.org/draft-04/schema#",
    "properties": {
        "username": {
            "title": "User name",
            "description": "Type the user name.",
            "type": "string"
        },
        "key": {
            "title": "Access key",
            "description": "Type your access key.",
            "type": "string"
        }
    },
    "required": ["username", "key"],
    "form": ["username", "key"]
}

Apologies if this sounded a bit preachy but I wanted to at least provide a few thoughts based on what we've learned. We're keen to work towards standardization in this space and share what we have here.

[mattmcneeney]

Thank you @skaegi for your detailed feedback. It's very helpful to hear from a team who have been using our proposed implementation for a few years and that your comments regarding its limitations seem to be consistent with the other feedback we have heard.

I think our plan for the first release of this will be supporting the core JSON schema keywords only, and that any possible extensions will be discussed in the future when we have strong and consistent use cases for them.

[bmelville]

We use JSONSchema in several products, and in some of the infrastructure that supports the brokers we're building. All support draft v4, and typically use language-specific validation libraries that support draft v3 and draft v4.

None of the tooling we've found even talks about draft v6, so I would be very hesitant adopting it as the exclusive schema standard until language specific tooling is available.

Most validators support all supported drafts, and if we require that schemas specify their version, we could support any/all supported supported versions.

My proposal would be at the minimum to support v4 initially or both v4 and v6 to allow people to utilize this now.

[mattmcneeney]

#165 is merged - closing this issue! Well done all!

[bmelville]

Huge 👍 for all the people who put in all the time to make this one a success.