proposal: encoding/json, encoding/xml: support zero values of structs with omitempty

[joeshaw]

Support zero values of structs with omitempty in encoding/json and encoding/xml.

This bites people a lot, especially with time.Time. Open bugs include #4357 (which has many dups) and #10648. There may be others.

Proposal

Check for zero struct values by adding an additional case to the isEmptyValue function:

case reflect.Struct:
        return reflect.Zero(v.Type()).Interface() == v.Interface()

This will solve the vast majority of cases.

(Optional) Introduce a new encoding.IsZeroer interface, and use this to check for emptiness:

Update: I am dropping this part of the proposal, see below.

type IsZeroer interface {
        IsZero() bool
}

Visit this playground link and note that the unmarshaled time.Time value does not have a nil Location field. This prevents the reflection-based emptiness check from working. IsZero() already exists on time.Time, has the correct semantics, and has been adopted as a convention by Go code outside the standard library.

An additional check can be added to the isEmptyValue() functions before checking the value's Kind:

if z, ok := v.Interface().(encoding.IsZeroer); ok {
        return z.IsZero()
}

Compatibility

The encoding.IsZeroer interface could introduce issues with existing non-struct types that may have implemented IsZero() without consideration of omitempty. If this is undesirable, the encoding.IsZeroer interface check could be moved only within the struct case:

case reflect.Struct:
        val := v.Interface()
        if z, ok := val.(encoding.IsZeroer); ok {
                return z.IsZero()
        }
        return reflect.Zero(v.Type()).Interface() == val

Otherwise, this change is backward-compatible with existing valid uses of omitempty. Users who have applied omitempty to struct fields incorrectly will get their originally intended behavior for free.

Implementation

I (@joeshaw) have implemented and tested this change locally, and will send the CL when the Go 1.6 tree opens.

[gopherbot]

CL https://golang.org/cl/13914 mentions this issue.

[gopherbot]

CL https://golang.org/cl/13977 mentions this issue.

[joeshaw]

The empty struct approach is implemented in CL 13914 and the IsZeroer interface is implemented in CL 13977.

In order for them to be reviewable separately they conflict a bit -- mostly in the documentation -- but I will fix for one if the other is merged.

[joeshaw]

In the CLs @rsc said,

I'd really like to stop adding to these packages. I think we need to leave well enough alone at some point.

I see what he's getting at. CL 13977, which implements the IsZeroer interface is clearly an enhancement and adds API to the standard library that needs to be maintained forever. So, I am abandoning that CL and that part of the proposal.

However, I still feel strongly about omitempty with empty structs, and I want to push for CL 13914 to land for Go 1.6.

I use the JSON encoding in Go a lot, as my work is mostly writing services that communicate with other services, in multiple languages, over HTTP. The fact that structs don't obey omitempty is a frequent source of confusion (see #4357 and its many dups and references, and #10648) and working around it is really annoying. Other programming languages do not conform to Go's ideal "zero value" idea, and as a result encoding a zero value is semantically different in JSON than omitting it or encoding it as null. People run into this most commonly with time.Time. (There is also the issue that decoding a zero time.Time does not result in an empty struct, see #4357 (comment) for background on that.)

I think it should be considered a bug that Go does not support omitempty for these types, and although it adds a small amount of additional code, it fixes a bug.

[jeromenerf]

This proposal is marked as unplanned, yet the related bug report #10648 is marked as go1.7.
Is it still being worked /thought on?

[rsc]

To my knowledge, it is not being worked on. Honestly this seems fairly low
priority and will likely miss Go 1.7.

On Sun, Mar 27, 2016 at 12:22 PM Jérôme Andrieux [email protected]
wrote:

This proposal is marked as unplanned, yet the related bug report #10648
#10648 is marked as go1.7.
Is it still being worked /thought on?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#11939 (comment)

[jeromenerf]

OK.

This is more of a convenience than a priority indeed.

It can be a pain point when dealing with libs that don't support "embedded structs" as pointer though.

[Perelandric]

I wonder if a low-impact alternative to the IsZeroer interface would be to allow one to return an error called json.CanOmit (or similar) from an implementation of the Marshaler interface. That way the dev is in control of determining what constitutes a zero value, and it doesn't impact other code.

It's not a perfect solution, since one can't add methods to types defined in another package, but this can be worked around to a degree.

Taking the time.Time example:

type MyTime struct {
  time.Time
}

// Implement the Marshaler interface
func (mt MyTime) MarshalJSON() ([]byte, error) {
  res, err := json.Marshal(mt.Time)

  if err == nil && mt.IsZero() {
    return res, json.CanOmit // Exclude zero value from fields with `omitempty`
  }
  return res, err
}

I haven't looked into implementation, but on the surface it would seem like a low-overhead solution, assuming the only work would be to check if an error returned equals json.CanOmit on fields where omitempty was included.

Using errors as a flag is not without precedent in the standard library, e.g. filepath#WalkFunc allows one to return filepath.SkipDir to skip recursion into a directory.

[joeshaw]

@Perelandric I mentioned a possible sentinel error value in https://golang.org/cl/13914 but I didn't get feedback on the idea or an opportunity to implement it before the Go 1.7 freeze. After Russ's comments on my original CL (and showing the unexpected difficulty in implementing this) I think that's the better way to go.

[gopherbot]

CL https://golang.org/cl/23088 mentions this issue.

[adg]

While it's clear that we can do this, it's not clear that we want to. I'd like to see a formal proposal document that weighs the advantages and drawbacks of this feature addition. In particular, I am concerned about compatibility and maintenance burden.

[joeshaw]

thanks Andrew. I worked on this a little bit at GopherCon. I will look into putting together a formal proposal.

[albrow]

@joeshaw we ran into this issue at my place of work and I'm eagerly awaiting your proposal. Feel free to contact me if you would like any help. Email is on my profile.

[Perelandric]

@joeshaw Is the proposal you're considering based on the sentinel object idea, or are you considering a different approach? Do you think you'll have time for this before the next release?

[joeshaw]

@Perelandric Yes, I think the sentinel object idea is the most straightforward way to go.

Other options include:

• The IsZeroer interface (but I think this has potential backward compatibility issues)
• "Backspacing" over objects that serialize to {} (but I think this requires too big a change to the JSON encoder code, and doesn't handle json.Marshaler implementers like time.Time)

I don't think I will be able to do this (proposal + implementation) before Go 1.8. If someone else wants to take it on for 1.8, I will gladly pass along my knowledge and partial implementation.

[Perelandric]

Thanks @joeshaw. I created an implementation using the sentinel error for the encoding/json package and will start to work on the proposal in a bit. I think I'll focus primarily on this approach.

The Marshaler interface in encoding/xml is different from that in encoding/json, and seems as though a custom zero-value can already be established without needing to return anything special. Did you find that to be true?

After I make a little more progress, I'll post a link to a branch in case you, @albrow or anyone else wishes to review and contribute.

If you have any additional thoughts or info in the meantime, please let me know. Thank you!

[Perelandric]

Change of heart on this. If there's resistance to adding to packages, then this won't fly. Maybe someone else wishes to advocate for this.

[pschultz]

Mentioned in one of the duplicate issues was the idea to let MarshalJSON return (nil, nil) to skip the field. Borrowing your earlier example:

type MyTime struct { time.Time }

func (mt MyTime) MarshalJSON() ([]byte, error) {
    if mt.IsZero() {
        return nil, nil // Exclude zero value from fields with `omitempty`
    }

    return json.Marshal(mt.Time)
}

In Go 1.7, returning nil is not a valid implementation for MarshalJSON and leads to "unexpected end of JSON input" errors. This approach doesn't require any visible change to the encoding package (not even adding an error value).

For what it's worth, I just intuitively wrote a MarshalJSON method like that, expecting a field to be omitted from the JSON output.

[joeshaw]

@pschultz That approach seems reasonable to me, but it can't be used with omitempty.

The reason you get that error is because the JSON encoder checks for the validity of the JSON coming out of MarshalJSON and the result of returning a nil byte slice is (something like) "key":,. If returning a nil byte slice indicated that it should be omitted that'd be a different way to omit something from being encoded in the JSON than omitempty. (That might be fine, it seems ok to me.)

The benefit of the error value is that it could fit in with the existing omitempty because you'd return something like []byte(""), ErrOmitEmpty and it'd obey omitempty yet still return a valid value ("") if not set.

[Perelandric]

@pschultz: A nil return wouldn't be able to be used as a flag for omitempty without causing undesired errors when omitempty is not present, since the implementation of MarshalJSON doesn't know when omitempty is actually there.

I don't know if having nil as an alternative to omitempty would be the best either. Seems like the user of a type should be the one deciding when it is omitted. IMO, the implementation of MarshalJSON should always return some useful representation, or an error when that's impossible.

[ianlancetaylor]

There are some slow-moving plans for encoding/json/v2.

[dsnet]

Hi all, we kicked off a discussion for a possible "encoding/json/v2" package that addresses the spirit of this proposal at least for JSON. See the "omitzero" struct tag option under the "Struct tag options" section.

[thw0rted]

My team inherited maintenance duties for a Go application, and we just had to fix a bug caused by the legacy code trying to use the omitempty tag on a nested (inline) struct member for JSON serialization. I've read a lot of related issues here, and started digging through this one, but Github is terrible at presenting a large number of comments on a single issue. I'm trying to understand the current state of things -- if there's a good summary somewhere in the middle of the ~200 comments on this issue, I'd appreciate a link.

As a JS developer, I can tell you that sending an empty object value ({}) over the wire is, to a first approximation, never the desired outcome. I could point to a decade (!) of issues where people are trying to emit sensible JSON. From an outsider's perspective, "all" it would take to accomplish this is to make the existing keyword do what it says on the tin, instead of having no effect (!) -- we just want the package to follow the Principle of Least Surprise. It sounds like there is some implementation detail that makes avoiding empty-object output difficult, or would incur an unacceptable performance penalty. Personally, I would rather have good output slowly than bad output fast.

[nyetwurk]

As a workaround, I've had good luck using https://github.com/clarketm/json (which is a re-implementation of encoding/json that omits empty structs) and using a type alias to prevent recursion

import (
	"encoding/json"

	jsonx "github.com/clarketm/json"
)

func (s OuterStruct) MarshalJSON() ([]byte, error) {
	// Use a type Alias to prevent Marshal recursion
	type TmpStruct OuterStruct
	// jsonx omits empty structs
	return jsonx.Marshal(TmpStruct(s))
}

[joeshaw]

The omitzero proposal in #45669 is pretty similar to the original proposal, and should cover the main use cases, so I am going to close this issue. A new issue will need to be raised for encoding/xml but it would be good for that to be its own issue.

[thw0rted]

If this is being closed in favor of #45669, is there a separate issue already going that I could follow to make applying omitempty to nested struct members -- which, if I understand correctly, can never do anything -- emit a compiler error, or at least a gopls warning?

[dsymonds]

That sounds like a reasonable `govet` enhancement.

[thw0rted]

Thanks @dsymonds , do you know where it would be appropriate to make that request? I can't tell if govet is managed in this repo or under https://github.com/golangci/golangci-lint -- I did find a https://github.com/golangci/govet but it was archived last year with no notes about why or where to take govet issues...

ETA: I found https://pkg.go.dev/golang.org/x/[email protected]/go/analysis/passes/structtag which appears to implement the "analysis" suite checker for struct tags specifically. That info page doesn't link to an issue tracker, as far as I can tell.

ETA again: apparently I commented on #51261 previously, which is a very similar proposal but currently only scoped to time.Time values, rather than any zero-value struct. I don't know whether the maintainers would prefer a second issue, or whether it would be better to broaden the scope of the existing one. (Personally I prefer the latter, but it's not my project, so...)

[dsymonds]

@thw0rted: This issue tracker is probably fine. There's plenty of other issues for govet already here.

[ianlancetaylor]

A new vet check should be opened in this issue tracker as a proposal. See https://go.dev/s/proposal. Thanks.

[thw0rted]

@ianlancetaylor it looks like the candidate implementation for #51261 actually covers this case, even though the title of that proposal still says it's only about time.Time. Does a new proposal still make sense in that case, compared to updating the scope of the existing one?

[ianlancetaylor]

Thanks, I agree that the existing proposal is sufficient here.