User login scoping (API) #20506

NickLaMuro · 2020-09-01T01:20:59Z

NOTE: Currently built on top of #20471

~~(though, this probably could be done without, but worth noting the Benchmarks included those changes)~~

A collection of tweaks to the User and Authenticator::* models to enable more effecient querying of User and related models

Requires manageiq-api changes to take effect: ManageIQ/manageiq-api#888

Benchmarks

Note: Improvements are in number of queries.

Before

$ bundle exec miqperf benchmark -ac2 "/api/users"
$ bundle exec miqperf report --last
/api/auth
|   ms | queries | query (ms) | rows |
| ---: |    ---: |       ---: | ---: |
| 5088 |      26 |        183 |   10 |
/api/users
|   ms | queries | query (ms) | rows |
| ---: |    ---: |       ---: | ---: |
|  267 |      12 |         13 | 1494 |
|   17 |      10 |        3.9 |   12 |

After

$ bundle exec miqperf benchmark -ac2 "/api/users"
$ bundle exec miqperf report --last
/api/auth
|   ms | queries | query (ms) | rows |
| ---: |    ---: |       ---: | ---: |
| 4949 |      12 |        181 |    1 |
/api/users
|   ms | queries | query (ms) | rows |
| ---: |    ---: |       ---: | ---: |
|  258 |       7 |         12 | 1489 |
|   15 |       5 |        3.3 |    7 |

Links

User/Role/Feature lookup improvements manageiq-api#888
More improvements as part of this effort: API Performance manageiq-api#880

kbrock

very nice changes

I am a little concerned about caching users and keeping them in memory for development purposes.
Also curious how this would do in production.

there are probably not very many scopes, so in terms of memory, this seems fine to me. I do wonder about storing default scope for almost every user. seems that a fallback may be the same without all the scope storage

A little confused what the first level of lookup scope buys us. But that is probably because I don't see too many examples of this besides the api.

app/models/user.rb

kbrock · 2020-09-02T19:09:29Z

app/models/authenticator/base.rb

- def lookup_by_identity(username, *_args)
- case_insensitive_find_by_userid(username)
+ def lookup_by_identity(username, request: nil, lookup_scope: nil)
+ @lookup ||= {}


what is the lifespan of this cache?

do we need to bother with a timeout mechanism?
or will it break code reloading in development?

or could we pass the hash in? not sure how that would worm

what is the lifespan of this cache?

Just for the duration of the request. This method was called multiple times as part of the Authenticator::Base#authenticate method, so was hoping to avoid redundant lookups. That said, I put it in a "hash" incase it was ever passed different args, and I didn't want to return an invalid cached result if it wasn't the same as was requested originally.

app/models/authenticator/base.rb

kbrock · 2020-09-02T19:14:23Z

app/models/authenticator/base.rb

@@ -200,8 +199,10 @@ def authenticate_with_http_basic(username, password, request = nil, options = {}
 [!!result, username]
 end

- def lookup_by_identity(username, *_args)
- case_insensitive_find_by_userid(username)
+ def lookup_by_identity(username, request: nil, lookup_scope: nil)


I am not sure the advantage of this method.
do both this class and case_insensitive_find use the same caching lookup?
Think they can be merged? (I'm probably not seeing a character change or something in there)

No, I was kinda confused about this as well. I decided not to address that in this PR.

let me try again.
you added a cache here
you added a cache in the case_insensitive_find_by_userid

can we get away with making this method just the call into case_insensitive_find_by_userid ?

lol. ^ idiot. totally didn't process your response.

LGTM

app/models/user.rb

Fryguy · 2020-10-12T16:44:39Z

app/models/user.rb

+ # Where :default_scoped is the value for anything that doesn't match a key in
+ # the original array. The "lookup hash" was chossen since it is much faster
+ # to do a hash table lookup than a `Array.includes?(KEY)`.
+ LOOKUP_SCOPES = [:api_includes].inject(Hash.new { :default_scoped }) {|h,i| h[i] = i; h}.freeze


I think this is a little cleaner, personally...wdyt?

LOOKUP_SCOPES = Hash.new(:default_scoped).merge([:api_includes].index_by(&:itself))

Or, if you don't really need the Array thing...

LOOKUP_SCOPES = Hash.new(:default_scoped).merge(:api_includes => :api_includes)

app/models/user.rb

Fryguy · 2020-10-12T16:48:22Z

Apologies for the late reply...was actually looking at the follow up PRs and realized I never commented on the base PR.

NickLaMuro · 2020-10-12T16:55:23Z

Apologies for the late reply...

@Fryguy no worries, though this is probably going to take a back seat since I don't really have a need for this at the moment, and I pushed it off into ManageIQ/manageiq-api#889 as well.

Also... I really don't want to bother with rebasing it at the moment...

kbrock

I'm a big fan of these changes.

Looks like the failure may be legit though

NickLaMuro · 2021-01-25T15:42:57Z

Okay, I wouldn't mind waiting until CI does pass, but I think this failure is from Libor's changes (and by extension, @kbrock ):

https://travis-ci.com/github/ManageIQ/manageiq/jobs/474546624#L791-L802

So I will wait to see if master starts passing before I try again.

A scope for eager loading `current_user` associated records that will be used in the process of authenticating on every request. Will be used in subsequent commits.

kbrock

think most of the rubocop complaints are that we are using colon hashes instead of hash rockets.

app/models/user.rb

kbrock · 2021-02-05T18:29:33Z

you going to address all the hashrockets - or you want to ship as is

ruby 3.0 does need colons and not hash rockets

And applies them to the `Authenticator` models. This allows components of the application (like the API) to pass specific scopes for the user that is being authenticated to include so they can be loaded in one fell swoop, but only ones we allow via the LOOKUP_SCOPES constant. This also changes a lot of the `lookup_by_*` methods to use kwargs to simplify the interface a bit, and support the new `:lookup_scope` argument.

miq-bot · 2021-02-05T19:30:49Z

Checked commits NickLaMuro/manageiq@54b48d3~...bb58c70 with ruby 2.6.3, rubocop 0.82.0, haml-lint 0.35.0, and yamllint
6 files checked, 23 offenses detected

app/models/authenticator/base.rb

⚠️ - Line 203, Col 38 - Lint/UnusedMethodArgument - Unused method argument - request.
❗ - Line 204, Col 49 - Style/HashSyntax - Use hash rockets syntax.
❗ - Line 68, Col 101 - Style/HashSyntax - Use hash rockets syntax.
❗ - Line 68, Col 119 - Style/HashSyntax - Use hash rockets syntax.
❗ - Line 79, Col 59 - Style/HashSyntax - Use hash rockets syntax.
❗ - Line 79, Col 77 - Style/HashSyntax - Use hash rockets syntax.

app/models/authenticator/database.rb

⚠️ - Line 13, Col 43 - Lint/UnusedMethodArgument - Unused method argument - request.
❗ - Line 14, Col 56 - Style/HashSyntax - Use hash rockets syntax.

app/models/authenticator/httpd.rb

⚠️ - Line 44, Col 59 - Lint/UnusedMethodArgument - Unused method argument - lookup_scope.
❗ - Line 101, Col 53 - Style/HashSyntax - Use hash rockets syntax.
❗ - Line 92, Col 51 - Style/HashSyntax - Use hash rockets syntax.
❗ - Line 93, Col 64 - Style/HashSyntax - Use hash rockets syntax.
❗ - Line 95, Col 57 - Style/HashSyntax - Use hash rockets syntax.

app/models/authenticator/ldap.rb

⚠️ - Line 69, Col 43 - Lint/UnusedMethodArgument - Unused method argument - request.
⚠️ - Line 69, Col 57 - Lint/UnusedMethodArgument - Unused method argument - lookup_scope.

app/models/user.rb

❗ - Line 286, Col 58 - Style/HashSyntax - Use hash rockets syntax.
❗ - Line 72, Col 24 - Style/Lambda - Use the lambda method for multiline lambdas.
❗ - Line 74, Col 7 - Layout/FirstArrayElementIndentation - Use 2 spaces for indentation in an array, relative to the first position after the preceding left parenthesis.
❗ - Line 77, Col 5 - Layout/FirstArrayElementIndentation - Indent the right bracket the same as the first position after the preceding left parenthesis.

spec/models/authenticator/httpd_spec.rb

❗ - Line 110, Col 50 - Style/HashSyntax - Use hash rockets syntax.
❗ - Line 114, Col 53 - Style/HashSyntax - Use hash rockets syntax.
❗ - Line 118, Col 53 - Style/HashSyntax - Use hash rockets syntax.
❗ - Line 122, Col 48 - Style/HashSyntax - Use hash rockets syntax.

NickLaMuro · 2021-02-05T19:39:14Z

you going to address all the hashrockets - or you want to ship as is

@kbrock All of these are for method arguments that are kwargs, which is required. As Jason said in gitter:

I've always said that the arg: value syntax made sense for keyword args , and it should not have been reused for hashes 😉

and I agree. Also, as you said (I guess...)

ruby 3.0 does need colons and not hash rockets [for kwargs]

Which is the only place these rubocop warnings are from, so they are irrelevant.

This is also a style for kwargs that is used elsewhere in the application:

manageiq/lib/ansible/runner.rb

Line 15 in 72fa6ef

 def run_async(env_vars, extra_vars, playbook_path, hosts: ["localhost"], credentials: [], verbosity: 0, become_enabled: false) 

So I am not doing anything non-standard here.

Fryguy · 2021-02-11T18:01:11Z

I wonder if rubocop has a way to differentiate Hashes and keyword args (particularly because of the upcoming 3.0 changes)

miq-bot · 2021-04-22T16:15:54Z

This pull request is not mergeable. Please rebase and repush.

miq-bot · 2023-02-27T00:19:39Z

This pull request has been automatically closed because it has not been updated for at least 3 months.

Feel free to reopen this pull request if these changes are still valid.

Thank you for all your contributions! More information about the ManageIQ triage process can be found in the triage process documentation.

Fryguy · 2023-07-27T15:39:18Z

cc @kbrock

miq-bot · 2023-07-28T17:40:46Z

This pull request is not mergeable. Please rebase and repush.

miq-bot · 2023-10-30T00:00:08Z

This pull request has been automatically closed because it has not been updated for at least 3 months.

Feel free to reopen this pull request if these changes are still valid.

Thank you for all your contributions! More information about the ManageIQ triage process can be found in the triage process documentation.

NickLaMuro requested review from agrare, Fryguy and kbrock as code owners September 1, 2020 01:20

NickLaMuro mentioned this pull request Sep 1, 2020

User/Role/Feature lookup improvements ManageIQ/manageiq-api#888

Open

gtanzillo mentioned this pull request Sep 1, 2020

API Performance ManageIQ/manageiq-api#880

Closed

8 tasks

kbrock reviewed Sep 2, 2020

View reviewed changes

NickLaMuro force-pushed the user-login-scoping branch from b8e8f2e to 06c71cc Compare September 2, 2020 22:07

chessbyte assigned kbrock Sep 3, 2020

chessbyte added the performance label Sep 3, 2020

NickLaMuro mentioned this pull request Sep 28, 2020

API Performance (Round 2) ManageIQ/manageiq-api#889

Open

4 tasks

Fryguy reviewed Oct 12, 2020

View reviewed changes

app/models/user.rb Outdated Show resolved Hide resolved

miq-bot added the unmergeable label Jan 14, 2021

NickLaMuro force-pushed the user-login-scoping branch from 06c71cc to 14bf42f Compare January 15, 2021 21:06

miq-bot removed the unmergeable label Jan 16, 2021

NickLaMuro force-pushed the user-login-scoping branch 2 times, most recently from 8f55dae to 734234b Compare January 22, 2021 21:56

kbrock reviewed Jan 23, 2021

View reviewed changes

NickLaMuro force-pushed the user-login-scoping branch from 734234b to f0d7cb8 Compare January 23, 2021 01:06

NickLaMuro requested a review from gtanzillo as a code owner January 23, 2021 01:06

[User] Add api_includes

54b48d3

A scope for eager loading `current_user` associated records that will be used in the process of authenticating on every request. Will be used in subsequent commits.

NickLaMuro force-pushed the user-login-scoping branch from f0d7cb8 to e00448c Compare January 25, 2021 17:41

kbrock reviewed Feb 3, 2021

View reviewed changes

app/models/user.rb Outdated Show resolved Hide resolved

NickLaMuro added 2 commits February 5, 2021 13:28

[Authenticator::Base] Add lookup cache

bb58c70

NickLaMuro force-pushed the user-login-scoping branch from e00448c to bb58c70 Compare February 5, 2021 19:28

miq-bot added the unmergeable label Apr 22, 2021

miq-bot added the stale label Feb 27, 2023

miq-bot closed this Feb 27, 2023

Fryguy reopened this Jul 27, 2023

Fryguy removed the stale label Jul 27, 2023

miq-bot added the stale label Oct 30, 2023

miq-bot closed this Oct 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

User login scoping (API) #20506

User login scoping (API) #20506

NickLaMuro commented Sep 1, 2020 •

edited

Loading

kbrock left a comment

kbrock Sep 2, 2020

NickLaMuro Sep 2, 2020

kbrock Sep 2, 2020

NickLaMuro Sep 2, 2020

kbrock Sep 8, 2020

kbrock Dec 14, 2020

Fryguy Oct 12, 2020 •

edited

Loading

Fryguy Oct 12, 2020 •

edited

Loading

Fryguy commented Oct 12, 2020

NickLaMuro commented Oct 12, 2020

kbrock left a comment

NickLaMuro commented Jan 25, 2021

kbrock left a comment

kbrock commented Feb 5, 2021

miq-bot commented Feb 5, 2021

NickLaMuro commented Feb 5, 2021 •

edited

Loading

Fryguy commented Feb 11, 2021

miq-bot commented Apr 22, 2021

miq-bot commented Feb 27, 2023

Fryguy commented Jul 27, 2023

miq-bot commented Jul 28, 2023

miq-bot commented Oct 30, 2023

User login scoping (API) #20506

User login scoping (API) #20506

Conversation

NickLaMuro commented Sep 1, 2020 • edited Loading

Benchmarks

Before

After

Links

kbrock left a comment

Choose a reason for hiding this comment

kbrock Sep 2, 2020

Choose a reason for hiding this comment

NickLaMuro Sep 2, 2020

Choose a reason for hiding this comment

kbrock Sep 2, 2020

Choose a reason for hiding this comment

NickLaMuro Sep 2, 2020

Choose a reason for hiding this comment

kbrock Sep 8, 2020

Choose a reason for hiding this comment

kbrock Dec 14, 2020

Choose a reason for hiding this comment

Fryguy Oct 12, 2020 • edited Loading

Choose a reason for hiding this comment

Fryguy Oct 12, 2020 • edited Loading

Choose a reason for hiding this comment

Fryguy commented Oct 12, 2020

NickLaMuro commented Oct 12, 2020

kbrock left a comment

Choose a reason for hiding this comment

NickLaMuro commented Jan 25, 2021

kbrock left a comment

Choose a reason for hiding this comment

kbrock commented Feb 5, 2021

miq-bot commented Feb 5, 2021

NickLaMuro commented Feb 5, 2021 • edited Loading

Fryguy commented Feb 11, 2021

miq-bot commented Apr 22, 2021

miq-bot commented Feb 27, 2023

Fryguy commented Jul 27, 2023

miq-bot commented Jul 28, 2023

miq-bot commented Oct 30, 2023

NickLaMuro commented Sep 1, 2020 •

edited

Loading

Fryguy Oct 12, 2020 •

edited

Loading

Fryguy Oct 12, 2020 •

edited

Loading

NickLaMuro commented Feb 5, 2021 •

edited

Loading