Security improvements in the Iceberg REST specification

[snazy]

Proposed Change

Following up on the mailing list discussion, we propose the following changes to Apache Iceberg. The following summary chapter is a summary of the initial message on this topic on the iceberg-dev mailing list.

We think that the ‘/v1/oauth/tokens’ endpoint in the Iceberg REST spec poses potential security and OAuth2 compliance issues, and excessively restricts how authorization should be implemented.

• As an open table format, it would be good for Iceberg to focus on the table format / catalog and not how authorization is implemented. The existence of an OAuth endpoint pushes implementations to adopt authorization using only OAuth, whereas the implementers might choose several other ways to implement authorization (e.g. SAML). In our opinion the spec should leave it open to the implementation to decide how authorization will be implemented.
• The existence of that endpoint also pushes operators of Iceberg REST endpoints into the authorization service business.
  Clients might expose their clear-text credentials to the wrong service, if the (correct) OAuth endpoint is not configured (humans do make mistakes).
• (Naive) Iceberg REST servers may proxy requests received for ‘/v1/oauth/tokens’ - and effectively become a “man-in-the-middle”, which is not fully compliant with the OAuth 2.0 specification.

The goals of this proposal are:

1. Secure the Iceberg REST specification by preventing accidental misuse/misimplementation.
2. Prevent that Iceberg REST to get into dictating the “authorization server specifics”.
3. Enable flexibility for Iceberg REST servers to opt for other authorization mechanisms than OAuth 2.0.
4. Enable REST servers to opt for integrating with any standard OAuth2 / OIDC provider (e.g. Okta, Keycloak, Authelia).

Proposed "milestones" are:
M1: Deprecate the /v1/oauth/tokens endpoint, targeting Apache Iceberg 1.7.0
M2: Update & clarify documentation, asap
M3: Define a pluggable REST client authorization framework, before M4
M4: Reference client authorization implementation(s) in Java, targeting Iceberg 1.8.0 or 2.0
M5: Removal the /v1/oauth/tokens endpoint, targeting Iceberg 1.9.0 or 2.0

Details in the linked document.

Proposal document

https://docs.google.com/document/d/1Xi5MRk8WdBWFC3N_eSmVcrLhk3yu5nJ9x_wC0ec6kVQ/

Specifications

• Table
• View
• REST
• Puffin
• Encryption
• Other

[adutra]

Should we add this to the "Iceberg REST Catalog" milestone?

https://github.com/apache/iceberg/milestone/46

[ajantha-bhat]

Should we add this to the "Iceberg REST Catalog" milestone?

I have added now.

[ajantha-bhat]

There is also a weekly sync planned for REST catalog. We have one on next Monday here the second point in agenda is to discuss the same.
https://lists.apache.org/thread/yy2qjbwk5lzd1ro2opl8zvb3lflw1pd8

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[dimas-b]

I believe this issue is still relevant.

More specifically, M3 is addressed by #11844 (and possibly some follow-up PRs).

[adutra]

I can confirm, we are still trying to achieve the milestones outlined above.