feat: Adding literals #76
Conversation
|
@zeroshade this is a considerable amount of work, I really need to learn more about the internals of iceberg, mostly working to understand the metadata. Looks great, nothing stands out as issues to me. |
literals.go
Outdated
| return t, nil | ||
| case DateType: | ||
| tm := time.UnixMicro(int64(t)).UTC() | ||
| return DateLiteral(tm.Truncate(24*time.Hour).Unix() / int64((time.Hour * 24).Seconds())), nil |
There was a problem hiding this comment.
Maybe good to break this out into a helper class, so it can be re-used.
There was a problem hiding this comment.
Shifted this into a method on the Timestamp type. It now has a ToDate() Date method which does this and the cast calls that method.
literals.go
Outdated
|
|
||
| return TimeLiteral(val), nil | ||
| case TimestampType: | ||
| val, err := arrow.TimestampFromString(string(s), arrow.Microsecond) |
There was a problem hiding this comment.
In Iceberg we are quite strict around our timezone handling. A TimestampType should have no timestamp, TimestampTzType should have mentioned the timestamp explicitly. See https://github.com/apache/iceberg-python/blob/42afc439d362ef1b3dcff03a1ffd959bc0a399ca/pyiceberg/utils/datetime.py#L29-L33
There was a problem hiding this comment.
@zeroshade can you fix this one before merging?
There was a problem hiding this comment.
Fixed, I also added tests to verify the errors for casting to TimestampType with a timezone and casting to TimestampTzType without a timezone. Let me know if I misunderstood anything. Thanks!
|
@zeroshade Thanks! Can you fix the conflicts? |
|
updated and fixed the conflicts @Fokko! 😄 |
|
@zeroshade Awesome, let's get this in! 🚀 |
After about 5 or 6 iterations of playing with how to represent expressions to start working on scan planning, I've finally settled on something I like!!
Rather than a huge code dump, I've broken it out into smaller successive PRs so that it is easier to review. This PR contains the definitions for representing Literals, which will then get utilized in constructing BooleanExpressions and so on.