kachery or poweruser setting won't work with logging

[hagikent]

Describe the bug
Fiber QC capsule is adopting the "alternative approach" that utilizes kachery to store reference figs and directly pushing QCevals to DocDB.
https://github.com/AllenNeuralDynamics/aind-fip-qc-raw

If you do:

import logging
setup_logging("aind-fip-qc-raw", mouse_id=mouse_id, session_name = sessionname)

for logging stuff, it gives errors.

@jeromelecoq suspects this is conflicting with kachery or AIND-poweruser setting.

Tag: @dbirman @yosefmaru

To Reproduce
see above

Expected behavior

Screenshots

Desktop (please complete the following information):
CodeOcean
https://codeocean.allenneuraldynamics.org/capsule/2763902

Additional context
Add any other context about the problem here.

[dbirman]

@hagikent can you clarify what exactly the errors are? This isn't really related to the QC portal itself so it would be good to move it to the right repository.

[hagikent]

Correct, not the issue of QC scheme per se.
I don't think this belongs to any other repo, though... could be a feature request in CO.

@dyf suggested not using kachery in the "alternative workflow" in a long-run.

@jeromelecoq any insight how to fix/circumvent this? I assume you still want to log QC related errors/messages?

[dbirman]

@hagikent

Can you confirm that this is the error you were running into?

Run environment setup complete, running code...

+ python -u run_capsule.py
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/logging/config.py", line 563, in configure
    handler = self.configure_handler(handlers[name])
  File "/opt/conda/lib/python3.8/logging/config.py", line 744, in configure_handler
    result = factory(**kwargs)
  File "/opt/conda/lib/python3.8/site-packages/watchtower/__init__.py", line 265, in __init__
    self._ensure_log_group()
  File "/opt/conda/lib/python3.8/site-packages/watchtower/__init__.py", line 301, in _ensure_log_group
    self._idempotent_call("create_log_group", logGroupName=self.log_group_name)
  File "/opt/conda/lib/python3.8/site-packages/watchtower/__init__.py", line 325, in _idempotent_call
    method_callable(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/opt/conda/lib/python3.8/site-packages/botocore/client.py", line 1023, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.AccessDeniedException: An error occurred (AccessDeniedException) when calling the CreateLogGroup operation: User: arn:aws:sts::467914378000:assumed-role/aind-codeocean-power-user/blmgidfp-ibib-klel-bamg-ecpbegmkohgj@worker is not authorized to perform: logs:CreateLogGroup on resource: arn:aws:logs:us-west-2:467914378000:log-group:aind/internal_logs:log-stream: because no identity-based policy allows the logs:CreateLogGroup action

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "run_capsule.py", line 26, in <module>
    setup_logging("aind-fip-qc-raw", mouse_id="test_mouse", session_name = "test")
  File "/opt/conda/lib/python3.8/site-packages/aind_log_utils/log.py", line 150, in setup_logging
    logging.config.dictConfig(log_config)
  File "/opt/conda/lib/python3.8/logging/config.py", line 808, in dictConfig
    dictConfigClass(config).configure()
  File "/opt/conda/lib/python3.8/logging/config.py", line 570, in configure
    raise ValueError('Unable to configure handler '
ValueError: Unable to configure handler 'watchtower'

I'm testing in this minimal copy of your capsule to try to isolate the issue: https://codeocean.allenneuraldynamics.org/capsule/8304033/tree

[jeromelecoq]

This is a conflict between the user permission required for the logging with those for katchery. I think David recommended just not to use katchery. I have asked ahad to help there *Jérôme*

…

On Fri, Jan 3, 2025 at 11:06 AM Dan Birman ***@***.***> wrote: @hagikent <https://github.com/hagikent> Can you confirm that this is the error you were running into? Run environment setup complete, running code... + python -u run_capsule.py Traceback (most recent call last): File "/opt/conda/lib/python3.8/logging/config.py", line 563, in configure handler = self.configure_handler(handlers[name]) File "/opt/conda/lib/python3.8/logging/config.py", line 744, in configure_handler result = factory(**kwargs) File "/opt/conda/lib/python3.8/site-packages/watchtower/__init__.py", line 265, in __init__ self._ensure_log_group() File "/opt/conda/lib/python3.8/site-packages/watchtower/__init__.py", line 301, in _ensure_log_group self._idempotent_call("create_log_group", logGroupName=self.log_group_name) File "/opt/conda/lib/python3.8/site-packages/watchtower/__init__.py", line 325, in _idempotent_call method_callable(*args, **kwargs) File "/opt/conda/lib/python3.8/site-packages/botocore/client.py", line 569, in _api_call return self._make_api_call(operation_name, kwargs) File "/opt/conda/lib/python3.8/site-packages/botocore/client.py", line 1023, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.AccessDeniedException: An error occurred (AccessDeniedException) when calling the CreateLogGroup operation: User: ***@***.*** is not authorized to perform: logs:CreateLogGroup on resource: arn:aws:logs:us-west-2:467914378000:log-group:aind/internal_logs:log-stream: because no identity-based policy allows the logs:CreateLogGroup action The above exception was the direct cause of the following exception: Traceback (most recent call last): File "run_capsule.py", line 26, in <module> setup_logging("aind-fip-qc-raw", mouse_id="test_mouse", session_name = "test") File "/opt/conda/lib/python3.8/site-packages/aind_log_utils/log.py", line 150, in setup_logging logging.config.dictConfig(log_config) File "/opt/conda/lib/python3.8/logging/config.py", line 808, in dictConfig dictConfigClass(config).configure() File "/opt/conda/lib/python3.8/logging/config.py", line 570, in configure raise ValueError('Unable to configure handler ' ValueError: Unable to configure handler 'watchtower' — Reply to this email directly, view it on GitHub <#37 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AATAHT53APOQDMIIZJWYP732I3NT5AVCNFSM6AAAAABT6BG6M6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRZGY4DSNBRG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[dbirman]

I guess I just don't understand why there would be a conflict, kachery just uses some environment variables to store it's AWS keys. But it sounds like maybe this was discussed offline already?

[dbirman]

Jerome and I sorted this out. @yosefmaru it would be helpful to add the logging permissions/keys to the power-user and not just the regular codeocean-user role, that would solve this particular situation. More generally there needs to be a discussion about whether the alternate workflow should exist or not.

[dbirman]

And @hagikent to close out the point here: because your capsule is part of a pipeline that will generate a data asset at the end your QC figures should go into the results folder and be stored in the data asset. So you shouldn't be using the alternate workflow here.

[jeromelecoq]

And @hagikent to close out the point here: because your capsule is part of a pipeline that will generate a data asset at the end your QC figures should go into the results folder and be stored in the data asset. So you shouldn't be using the alternate workflow here.

Adding what I understood: you should take the input data to the capsule, copy it to the result folder and append your plots to it.

[hagikent]

Sounds good. I can work on it with Ahad while integrating it to the pipeline.