Skip to content

Commit

Permalink
refactor(syncer): refine transform function (#606)
Browse files Browse the repository at this point in the history
## What type of PR is this?

/kind refactor

## What this PR does / why we need it:

The current transform function has two capabilities: trimming to save
informer memory and influencing data stored in ES. This PR splits these
two capabilities into two stages, making the function definition clearer
and having the following advantages:

1. The first stage uses jsonPath instead of go template, which is easier
for users to use and less prone to errors, and ensures that the data in
the informer is a subset of the original object.

2. The second stage can use workqueue to retry retryable errors that
occur during the transformation process.

## Which issue(s) this PR fixes:

<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related
issues/tests in a comment and do not use `Fixes`_*
-->

Fixes #
  • Loading branch information
iamryanchia authored Sep 12, 2024
1 parent 6514721 commit f4c389a
Show file tree
Hide file tree
Showing 36 changed files with 2,791 additions and 66 deletions.
2 changes: 2 additions & 0 deletions pkg/kubernetes/apis/search/register.go
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,8 @@ func addKnownTypes(scheme *runtime.Scheme) error {
&SyncResourcesList{},
&TransformRule{},
&TransformRuleList{},
&TrimRule{},
&TrimRuleList{},
)
return nil
}
40 changes: 40 additions & 0 deletions pkg/kubernetes/apis/search/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,46 @@ type ResourceSyncRule struct {

// TransformRefName is the name of the TransformRule
TransformRefName string

// Trim defines the trimming strategy for the resources of the current type.
Trim *TrimRuleSpec `json:"trim,omitempty"`

// TrimRefName is the name of the TrimRule.
TrimRefName string `json:"trimRefName,omitempty"`
}

// +genclient
// +genclient:nonNamespaced
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// TrimRule defines the strategy of trimming k8s objects, which can save
// informer memory by discarding redundant fields.
type TrimRule struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`

Spec TrimRuleSpec `json:"spec,omitempty"`
}

type TrimRuleSpec struct {
// Retain specifies which fields should be retained after trimming.
Retain TrimRuleRetainFields `json:"retain,omitempty"`
}

type TrimRuleRetainFields struct {
// JSONPaths specifies the path of the field to be retained.
// For usage, please refer to https://kubernetes.io/docs/reference/kubectl/jsonpath/
JSONPaths []string `json:"jsonPaths,omitempty"`
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

type TrimRuleList struct {
metav1.TypeMeta `json:",inline"`

metav1.ListMeta `json:"metadata,omitempty"`

Items []TrimRule `json:"items"`
}

// +genclient
Expand Down
2 changes: 2 additions & 0 deletions pkg/kubernetes/apis/search/v1beta1/register.go
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ func addKnownTypes(scheme *runtime.Scheme) error {
&SyncResourcesList{},
&TransformRule{},
&TransformRuleList{},
&TrimRule{},
&TrimRuleList{},
)
metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
return nil
Expand Down
46 changes: 46 additions & 0 deletions pkg/kubernetes/apis/search/v1beta1/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,52 @@ type ResourceSyncRule struct {
// TransformRefName is the name of the TransformRule
// +optional
TransformRefName string `json:"transformRefName,omitempty"`

// Trim defines the trimming strategy for the resources of the current type.
// +optional
Trim *TrimRuleSpec `json:"trim,omitempty"`

// TrimRefName is the name of the TrimRule.
// +optional
TrimRefName string `json:"trimRefName,omitempty"`
}

// +genclient
// +genclient:nonNamespaced
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// TrimRule defines the strategy of trimming k8s objects, which can save
// informer memory by discarding redundant fields.
type TrimRule struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"` //nolint:tagliatelle

// +optional
Spec TrimRuleSpec `json:"spec,omitempty"`
}

type TrimRuleSpec struct {
// Retain specifies which fields should be retained after trimming.
// +optional
Retain TrimRuleRetainFields `json:"retain,omitempty"`
}

type TrimRuleRetainFields struct {
// JSONPaths specifies the path of the field to be retained.
// For usage, please refer to https://kubernetes.io/docs/reference/kubectl/jsonpath/
// +optional
JSONPaths []string `json:"jsonPaths,omitempty"`
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

type TrimRuleList struct {
metav1.TypeMeta `json:",inline"`

// +optional
metav1.ListMeta `json:"metadata,omitempty"` //nolint:tagliatelle

Items []TrimRule `json:"items"`
}

// +genclient
Expand Down
136 changes: 136 additions & 0 deletions pkg/kubernetes/apis/search/v1beta1/zz_generated.conversion.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

103 changes: 103 additions & 0 deletions pkg/kubernetes/apis/search/v1beta1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit f4c389a

Please sign in to comment.