Feature/sckan-324 - journey computation #381

D-GopalKrishna · 2024-12-10T16:24:25Z

[ONLY BONUS DEVELOPMENT] - Others are not part of this PR and are explained below:
a. I didn't not see any rows that went empty.
b. export of statement preview is not implemented in this PR. Perhaps we can change the name of the card @ddelpiano ?

Task Implementation description:
Refactor the journey computation

persist the journey in the db
re-compute this everytime we change the entities in path builder are changed - adapting PATCH request to save connectivity statements too.

Demo

2024-12-10.21-40-51.mp4

…nnectivityStatement model

…_entities fields in ConnectivityStatement model and adapt graph_service

…ViewSet to save associated connectivity_statement

Conflicts: backend/composer/api/views.py

backend/composer/models.py

backend/composer/api/views.py

backend/composer/migrations/0064_auto_20241208_2313.py

afonsobspinto · 2024-12-11T18:41:42Z

backend/composer/services/graph_service.py

-
+    vias = list(Via.objects.filter(connectivity_statement__id=connectivity_statement.id))
+    destinations = list(Destination.objects.filter(connectivity_statement__id=connectivity_statement.id))
+    origins = list(Origin.objects.filter(origins_relations__id=connectivity_statement.id).distinct())


question: Why did we change from:
origins = list(connectivity_statement.origins.all())
to
origins = list(Origin.objects.filter(origins_relations__id=connectivity_statement.id).distinct())

Reason is here in the code - https://github.com/MetaCell/sckan-composer/blob/feature/SCKAN-324/backend/composer/services/graph_service.py#L24

It says:

Exception has occurred: AttributeError (note: full exception trace is shown but execution is paused at: _run_module_as_main) 'AnatomicalEntity' object has no attribute 'name' File "/home/dgk/metacell/sckan-composer/backend/composer/services/graph_service.py", line 24, in generate_paths paths.extend(create_paths_from_origin(origin, vias, destinations, [(str(origin.id), origin.name, 0)], destination_layer)) File "/home/dgk/metacell/sckan-composer/backend/composer/services/graph_service.py", line 166, in compile_journey all_paths2 = generate_paths(origins, vias, destinations) File "/home/dgk/metacell/sckan-composer/backend/composer/migrations/0067_auto_20241218_0928.py", line 11, in update_journey_fields cs.journey_path = compile_journey(cs) File "/home/dgk/miniconda3/envs/composer/lib/python3.9/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards self.code(from_state.apps, schema_editor) File "/home/dgk/miniconda3/envs/composer/lib/python3.9/site-packages/django/db/migrations/migration.py", line 130, in apply operation.database_forwards( File "/home/dgk/miniconda3/envs/composer/lib/python3.9/site-packages/django/db/migrations/executor.py", line 252, in apply_migration state = migration.apply(state, schema_editor) File "/home/dgk/miniconda3/envs/composer/lib/python3.9/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards state = self.apply_migration( File "/home/dgk/miniconda3/envs/composer/lib/python3.9/site-packages/django/db/migrations/executor.py", line 135, in migrate state = self._migrate_all_forwards( File "/home/dgk/miniconda3/envs/composer/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 349, in handle post_migrate_state = executor.migrate( File "/home/dgk/miniconda3/envs/composer/lib/python3.9/site-packages/django/core/management/base.py", line 96, in wrapped res = handle_func(*args, **kwargs) File "/home/dgk/miniconda3/envs/composer/lib/python3.9/site-packages/django/core/management/base.py", line 448, in execute output = self.handle(*args, **options) File "/home/dgk/miniconda3/envs/composer/lib/python3.9/site-packages/django/core/management/base.py", line 402, in run_from_argv self.execute(*args, **cmd_options) File "/home/dgk/miniconda3/envs/composer/lib/python3.9/site-packages/django/core/management/__init__.py", line 440, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/home/dgk/miniconda3/envs/composer/lib/python3.9/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line utility.execute() File "/home/dgk/metacell/sckan-composer/backend/manage.py", line 18, in main execute_from_command_line(sys.argv) File "/home/dgk/metacell/sckan-composer/backend/manage.py", line 22, in <module> main() File "/home/dgk/miniconda3/envs/composer/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/dgk/miniconda3/envs/composer/lib/python3.9/runpy.py", line 197, in _run_module_as_main (Current frame) return _run_code(code, main_globals, None, AttributeError: 'AnatomicalEntity' object has no attribute 'name'

so when generating the path - we do have the origins, but not the origin.name - which is a property in AnatomicalEntity.

@afonsobspinto did you do the check with the API call?

This happens only when we do the migration - possibly something with - apps.get_model('composer', 'ConnectivityStatement') in the migration file (ref - 0067 auto migration file). Also if you notice this is only for the @properties and not for the fields.
So if you put a breakpoint to the above mentioned code, and then run the migration to db - this should cause the error.

Understood, during migrations django uses historical models that don't include python level constructs like @Property.
I think the new instruction works, but I would remove the .distinct(), I don't think it's needed in our context

Before removing it, I want to confirm again with you if you are okay to remove the distinct() here, the reason I had that is because unlike vias and destinations - which are FK, origins is a M2M.

The ManyToManyField in Django automatically ensures that the relationship between two models is unique. In other words, we know that we have a unique pair of ConnectivityStatement and AnatomicalEntity. Here's a simple test:

With that said, if we have the time I would apply the same suggestion I mentioned here:
#404 (comment)

…s with journey_path in ConnectivityStatement model

…g in graph_service

…to signals

…ave method

backend/composer/models.py

afonsobspinto

Looking better.

Would it be possible to also change the admin page to not show or keep the journey path as read only?

afonsobspinto · 2025-01-06T15:44:37Z

backend/composer/services/graph_service.py

+    journey_paths = get_journey_path_from_consolidated_paths(
+        consolidated_paths)
+    Via = apps.get_model('composer', 'Via')
+    vias = list(Via.objects.filter(


question: Why do we need to fetch the vias here? can't we rely entirely on the new journey paths attribute created?

I understand what you mean yes, this is only to get the len(vias) later with - 0<layer<len(vias) + 1]

here is the journey_paths

and in this case, len(vias from the CS) is 1.
[<Via: Via object (37)>]

where the value of that via is:

{'_state': <django.db.models.base.ModelState object at 0x7f006b7fd8e0>, 'id': 37, 'connectivity_statement_id': 13, 'type': 'AXON', 'order': 0, '_prefetched_objects_cache': {'anatomical_entities': <QuerySet [<AnatomicalEntity: submucosa>, <AnatomicalEntity: nose>]>, 'from_entities': <QuerySet [<AnatomicalEntity: pituitary gland>, <AnatomicalEntity: zone of skin>]>}}

I am unable to see the pattern to relate them. Do you think there is one @afonsobspinto that I might be missing?

If I understand the code correctly we don't need the vias count; we can just do something on the lines of:

destination_layer = path[-1][1] # Destination's layer via_names = ' via '.join([node for node, layer in path if 0 < layer < destination_layer])

which identifies the vias by looking for nodes with layers between origin (0) and destination (highest layer)

afonsobspinto · 2025-01-06T15:45:54Z

backend/composer/services/graph_service.py

    entities = []
+    Via = apps.get_model('composer', 'Via')
+    vias = list(Via.objects.filter(


Same question as https://github.com/MetaCell/sckan-composer/pull/381/files#r1904313466

…tAdmin

D-GopalKrishna added 3 commits December 10, 2024 21:45

SCKAN-324 - Add journey_description and journey_entities fields to Co…

ddd7d9c

…nnectivityStatement model

SCKAN-324 - Add migration to populate journey_description and journey…

22ba03f

…_entities fields in ConnectivityStatement model and adapt graph_service

SCKAN-324 - adapt partial_update method in ViaViewSet and Destination…

ea1297f

…ViewSet to save associated connectivity_statement

D-GopalKrishna requested review from afonsobspinto and ddelpiano December 10, 2024 16:24

Merge remote-tracking branch 'refs/remotes/origin/develop'

5b03c6d

Conflicts: backend/composer/api/views.py

afonsobspinto requested changes Dec 11, 2024

View reviewed changes

D-GopalKrishna added 5 commits December 18, 2024 12:56

Merge remote-tracking branch 'origin/develop' into feature/SCKAN-324

ae6bcb0

SCKAN-324 - refactor: replace journey_description and journey_entitie…

01436c0

…s with journey_path in ConnectivityStatement model

SCKAN-324 - refactor: simplify journey compilation and entity buildin…

b8a1d9e

…g in graph_service

SCKAN-324 - refactor: remove save entities logic from partial_update …

50978d1

…to signals

SCKAN-324 - refactor tests to adapt to new consolidated_path service

51b3838

D-GopalKrishna requested a review from afonsobspinto December 18, 2024 11:40

SCKAN-324 - cleanup - simplify the assignment of compile_journey in s…

b934de3

…ave method

afonsobspinto reviewed Dec 19, 2024

View reviewed changes

backend/composer/models.py Outdated Show resolved Hide resolved

D-GopalKrishna added 2 commits December 20, 2024 13:00

SCKAN-324 - do not save journey on every save of ConnectivityStatement

99d1238

SCKAN-324 refactor: move journey recompilation to signal handlers

466d8b3

afonsobspinto requested changes Jan 6, 2025

View reviewed changes

afonsobspinto reviewed Jan 6, 2025

View reviewed changes

SCKAN-324 add journey_path to readonly fields in ConnectivityStatemen…

cc18966

…tAdmin

D-GopalKrishna mentioned this pull request Jan 14, 2025

Feature/SCKAN-364 - Add statement preview row to the export + persist statement preview suffix and prefix #404

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/sckan-324 - journey computation #381

Feature/sckan-324 - journey computation #381

D-GopalKrishna commented Dec 10, 2024 •

edited by jira bot

Loading

afonsobspinto Dec 11, 2024

D-GopalKrishna Dec 18, 2024 •

edited

Loading

D-GopalKrishna Dec 20, 2024

afonsobspinto Jan 6, 2025

D-GopalKrishna Jan 14, 2025

afonsobspinto Jan 14, 2025

afonsobspinto Jan 14, 2025

afonsobspinto left a comment •

edited

Loading

afonsobspinto Jan 6, 2025

D-GopalKrishna Jan 14, 2025 •

edited

Loading

afonsobspinto Jan 14, 2025 •

edited

Loading

afonsobspinto Jan 6, 2025

Feature/sckan-324 - journey computation #381

Are you sure you want to change the base?

Feature/sckan-324 - journey computation #381

Conversation

D-GopalKrishna commented Dec 10, 2024 • edited by jira bot Loading

Choose a reason for hiding this comment

D-GopalKrishna Dec 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

afonsobspinto left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

D-GopalKrishna Jan 14, 2025 • edited Loading

Choose a reason for hiding this comment

afonsobspinto Jan 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

D-GopalKrishna commented Dec 10, 2024 •

edited by jira bot

Loading

D-GopalKrishna Dec 18, 2024 •

edited

Loading

afonsobspinto left a comment •

edited

Loading

D-GopalKrishna Jan 14, 2025 •

edited

Loading

afonsobspinto Jan 14, 2025 •

edited

Loading