Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[uss_qualifier] Centralize traceback.format_exception call #427

Merged
merged 3 commits into from
Dec 20, 2023

Conversation

BenjaminPelletier
Copy link
Member

As evidenced by #367, we call traceback.format_exception in a number of different places in our codebase to construct a stracktrace string. Prompted by some mysterious observed exceptions, this PR makes a single stacktrace_string function that everywhere else in the codebase references, and that function is the only place that calls format_exception (this would have made #367 easier had it already existed).

While trying to reproduce the issue, I also discovered misbehavior when returning a tuple from idempotent handlers so this PR fixes that issue as well.

Finally, the codebase information is printed to console to help debug problems like this in the future.

Observed logs, lightly redacted:

  2023-12-19 11:16:04.381 | INFO     | __main__:main:185 - ========== Running uss_qualifier for configuration file:///tmp/tmpx47759m2.yaml ==========
  2023-12-19 11:16:04.430 | INFO     | __main__:run_config:123 - Validating configuration...
  2023-12-19 11:16:07.310 | INFO     | __main__:run_config:153 - Executing test run
  2023-12-19 11:16:07.310 | INFO     | __main__:execute_test_run:75 - Instantiating resources
  2023-12-19 11:16:07.400 | INFO     | __main__:execute_test_run:78 - Computing signatures of inputs
  2023-12-19 11:16:07.402 | INFO     | __main__:execute_test_run:93 - Instantiating top-level test suite action
  2023-12-19 11:16:07.436 | WARNING  | monitoring.uss_qualifier.suites.suite:__init__:259 - Skipping action 0 (action_generator action_generators.astm.f3548.ForEachDSS) because Test suite action to run action_generator action_generators.astm.f3548.ForEachDSS could not find required resource ID "dss_instances" used to populate child resource ID "dss_instances"
  2023-12-19 11:16:07.437 | WARNING  | monitoring.uss_qualifier.suites.suite:__init__:259 - Skipping action 1 (test_scenario scenarios.astm.utm.PrepareFlightPlanners) because Test suite action to run test_scenario scenarios.astm.utm.PrepareFlightPlanners could not find required resource ID "mock_uss" used to populate child resource ID "mock_uss"
  2023-12-19 11:16:08.047 | WARNING  | monitoring.uss_qualifier.suites.suite:__init__:259 - Skipping action 3 (action_generator action_generators.flight_planning.FlightPlannerCombinations) because Test suite action to run action_generator action_generators.flight_planning.FlightPlannerCombinations could not find required resource ID "priority_preemption_flights" used to populate child resource ID "priority_preemption_flights"
  2023-12-19 11:16:08.302 | WARNING  | monitoring.uss_qualifier.suites.suite:__init__:259 - Skipping action 5 (action_generator action_generators.flight_planning.FlightPlannerCombinations) because Test suite action to run action_generator action_generators.flight_planning.FlightPlannerCombinations could not find required resource ID "mock_uss" used to populate child resource ID "mock_uss"
  2023-12-19 11:16:08.657 | INFO     | __main__:execute_test_run:96 - Running top-level test suite action
  2023-12-19 11:16:08.657 | INFO     | monitoring.uss_qualifier.suites.suite:_run_test_suite:175 - Beginning test suite ASTM F3548-21...
  2023-12-19 11:16:08.658 | INFO     | monitoring.uss_qualifier.suites.suite:_run_test_scenario:146 - Running "Validation of operational intents" scenario...
  2023-12-19 11:16:08.659 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:289 - Note: Tested USS -> REDACTED
  2023-12-19 11:16:12.854 | DEBUG    | monitoring.uss_qualifier.scenarios.scenario:record_query:371 - Queried PUT https://REDACTED/flight_plans/4646769d-1a3d-4414-9f9c-360e715f87eb -> 403 (2.5s); attributed participant REDACTED and type interuss.automated_testing.flight_planning.v1.UpsertFlightPlan
  2023-12-19 11:16:12.854 | INFO     | monitoring.uss_qualifier.scenarios.scenario:record_note:289 - Note: REDACTED -> Exception occurred during OpIntentValidator (<class 'TypeError'>: format_exception() missing 2 required positional arguments: 'value' and 'tb').
  Traceback (most recent call last):
    File "/app/monitoring/monitoring/uss_qualifier/resources/flight_planning/flight_planner.py", line 171, in request_flight
      resp = self.client.try_plan_flight(
    File "/app/monitoring/monitoring/monitorlib/clients/flight_planning/client_v1.py", line 104, in try_plan_flight
      return self._inject(
    File "/app/monitoring/monitoring/monitorlib/clients/flight_planning/client_v1.py", line 67, in _inject
      raise PlanningActivityError(
  monitoring.monitorlib.clients.flight_planning.client.PlanningActivityError: Attempt to plan flight returned status 403 rather than 200 as expected

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/app/monitoring/monitoring/uss_qualifier/scenarios/flight_planning/test_steps.py", line 253, in submit_flight_intent
      resp, query, flight_id = flight_planner.request_flight(
    File "/app/monitoring/monitoring/uss_qualifier/resources/flight_planning/flight_planner.py", line 175, in request_flight
      raise QueryError(str(e), e.queries)
  monitoring.monitorlib.fetch.QueryError: Attempt to plan flight returned status 403 rather than 200 as expected

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/app/monitoring/monitoring/uss_qualifier/suites/suite.py", line 150, in _run_test_scenario
      scenario.run(context)
    File "/app/monitoring/monitoring/uss_qualifier/scenarios/astm/utm/flight_intent_validation/flight_intent_validation.py", line 150, in run
      self._attempt_invalid()
    File "/app/monitoring/monitoring/uss_qualifier/scenarios/astm/utm/flight_intent_validation/flight_intent_validation.py", line 182, in _attempt_invalid
      validator.expect_not_shared()
    File "/app/monitoring/monitoring/uss_qualifier/scenarios/astm/utm/test_steps.py", line 93, in __exit__
      raise exc_val
    File "/app/monitoring/monitoring/uss_qualifier/scenarios/astm/utm/flight_intent_validation/flight_intent_validation.py", line 172, in _attempt_invalid
      submit_flight_intent(
    File "/app/monitoring/monitoring/uss_qualifier/scenarios/flight_planning/test_steps.py", line 262, in submit_flight_intent
      details=f"{str(e)}\n\nStack trace:\n{e.stacktrace}",
    File "/app/monitoring/monitoring/monitorlib/fetch/__init__.py", line 321, in stacktrace
      return "".join(traceback.format_exception(self))
  TypeError: format_exception() missing 2 required positional arguments: 'value' and 'tb'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/app/monitoring/monitoring/uss_qualifier/main.py", line 205, in <module>
      sys.exit(main())
    File "/app/monitoring/monitoring/uss_qualifier/main.py", line 188, in main
      exit_code = run_config(
    File "/app/monitoring/monitoring/uss_qualifier/main.py", line 154, in run_config
      report = execute_test_run(whole_config)
    File "/app/monitoring/monitoring/uss_qualifier/main.py", line 97, in execute_test_run
      report = action.run(context)
    File "/app/monitoring/monitoring/uss_qualifier/suites/suite.py", line 131, in run
      report = TestSuiteActionReport(test_suite=self._run_test_suite(context))
    File "/app/monitoring/monitoring/uss_qualifier/suites/suite.py", line 176, in _run_test_suite
      report = self.test_suite.run(context)
    File "/app/monitoring/monitoring/uss_qualifier/suites/suite.py", line 285, in run
      _run_actions(actions(), context, report)
    File "/app/monitoring/monitoring/uss_qualifier/suites/suite.py", line 328, in _run_actions
      action_report = action.run(context)
    File "/app/monitoring/monitoring/uss_qualifier/suites/suite.py", line 134, in run
      action_generator=self._run_action_generator(context)
    File "/app/monitoring/monitoring/uss_qualifier/suites/suite.py", line 187, in _run_action_generator
      _run_actions(self.action_generator.actions(), context, report)
    File "/app/monitoring/monitoring/uss_qualifier/suites/suite.py", line 328, in _run_actions
      action_report = action.run(context)
    File "/app/monitoring/monitoring/uss_qualifier/suites/suite.py", line 128, in run
      test_scenario=self._run_test_scenario(context)
    File "/app/monitoring/monitoring/uss_qualifier/suites/suite.py", line 161, in _run_test_scenario
      scenario.record_execution_error(e)
    File "/app/monitoring/monitoring/uss_qualifier/scenarios/scenario.py", line 501, in record_execution_error
      self._scenario_report.execution_error = ErrorReport.create_from_exception(e)
    File "/app/monitoring/monitoring/uss_qualifier/reports/report.py", line 205, in create_from_exception
      stacktrace="".join(traceback.format_exception(e)),
  TypeError: format_exception() missing 2 required positional arguments: 'value' and 'tb'

@BenjaminPelletier BenjaminPelletier marked this pull request as ready for review December 19, 2023 23:39
@BenjaminPelletier BenjaminPelletier merged commit a80c4a3 into interuss:main Dec 20, 2023
9 checks passed
@BenjaminPelletier BenjaminPelletier deleted the format-exception branch December 20, 2023 16:23
github-actions bot added a commit that referenced this pull request Dec 20, 2023
* Centralize call to format_exception

* Fix idempotency return value

* Print codebase information in console a80c4a3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants