Consider adding metadata to the entry that allows a registrar to self-identify

[azdagron]

We've discussed this before but I failed to find the issue so I thought I'd open a new one to keep the conversation going.

SPIRE Controller Manager tries to be authoritative for the workload entries it manages. In its current design, it expects to be the sole registrar operating against a given SPIRE Server cluster. If it discovers entries that it does not think should exist, it removes them. The fallout of this choice is that SPIRE Controller Manager cannot exist side-by-side other registrars.

There has been a slow but steady trickle of asks by the community to allow SPIRE Controller Manager to coexist with other registrars (including other SPIRE Controller Managers). The reasons are varied. The weight behind the ask is growing and it's something I'd like to support somehow.

Entries IDs or hints can be (ab)used for this purpose but that is not ideal.

In the past, we've thought about adding a specific field for this purpose (e.g. "registrar", "owner", "namespace") but haven't come to agreement on things like:

• who sets this field
• is it authoritative or advisory
• what relationship does it have with access control, if any

For my specific use case, a simple, opaque advisory field would be sufficient.

Thoughts??

Related issues on the controller manager:

• Gradual rollout / multicluster support spire-controller-manager#312
• others? (looking)

[kfox1111]

I've needed this functionality too. I've tested abusing hints for this purpose with the spire-controller-manager, and it was very easy to implement, but I agree its probably not a good long term strategy.

My thoughts on your questions are as follows:

If someone needs hard security around all of it, they should probably be using separate spire-server/controller-manager instances to guarantee security. If we wanted to support hard multi-tenancy within spire-server proper, that would be ok too, but substantially more work then what is being needed to solve this softer problem.

So, that in mind, I'm thinking something simple/advisory is better too.

• The field is set by the controller using it and filters based on it like hints currently work.
• It would be advisory to keep things simple.
• There would be no additional access control on it like there isn't for hints. It would just be a way to mark ownership of an entry so one controller doesn't stomp on anothers entries (or the users manual entries)

And things not filtering by entry owner would see all entries just as now.

As for implementation, I'm thinking it would basically mirror the work that was done for hints?

[StupidScience]

Will add my 5 cents here.
If it is going to be a metadata then you can look on how it is done in general in k8s with labels/annotations, e.g. it is key value string:string.

who sets this field

If it is metadata and not specific field anyone who has impact on the process of entry creation, e.g.

• whoever define clusterSpiffeId add some metadata for entry owners: owner team/person, service, email, etc
• Controller manager adds its own metadata, e.g. cluster, class, region, parent clusterSpiffeId, managedBy, etc
• Spire server can add its own metadata if necessary

is it authoritative or advisory

Rather advisory with “system” reserved keys or prefixes. E.g. controller manager can prefix all keys with controller-manager.spiffe.io -> controller-manager.spiffe.io/cluster

what relationship does it have with access control, if any

Not sure I get it. Perhaps if you have additional logic around it could be used for access controls, otherwise consider metadata as metadata and no direct effect on ACL.

I believe there is little to no difference of purposes with what Hint currently has, just could be more structured or hint could be considered as one of metadata fields.

[kfox1111]

If its a field and not a generic metadata property, like hint is, we can add a filter which entries get handed back to the controller for processing. This could save a lot of network trips for syncing to the registrar when multiple are involved. Like: https://github.com/spiffe/spire-api-sdk/blob/v1.8.7/proto/spire/api/server/entry/v1/entry.pb.go#L913

[StupidScience]

Have little experience with sql so correct if I’m wrong but I can imagine you can extract string:string metadata to separate table like entries_metadata with columns entryID, key, value and composite PK entryID+key. Then entries should be queriable by one or multiple matching keys. It can also help to differentiate if you need get just all queries or want to make more heavy query and include metadata.

Just not sure if adding specific field for each metadata that can be required for all potential use cases is sustainable approach.

[azdagron]

I'd wager generic entry metadata is likely out of scope as a solution for this request. Probably worth considering, but there has traditionally been a fair bit of opposition to the idea for a variety of reasons.

[StupidScience]

@azdagron any consensus on the approach to use here? Also what would be a timeline? I guess it won't be backported so we would expect to see it in at least 1.10?
Just there seems to be no "supported" way to migrate from removed in 1.6 k8s-registrar to new controller manager without disruptions.

[azdagron]

There isn't clear consensus on a path forward for first-class support of a concept like this in SPIRE. For the immediate needs of the spire-controller-manager, we've decided to use entry id prefixing as a stop-gap. If/when SPIRE opens up support for something different, we can leverage it then.

See spiffe/spire-controller-manager#325