refactor: use attributes #594

xgui3783 · 2024-05-14T15:36:43Z

supercedes #553

EDIT: The tests are turned off, except for the ones in examples_new to see if there is regression. The remaining test migration will be made gradually. (b70abe9 and 8a47f0f)

A list of issues this PR solves (Please add more moving forward):

xgui3783 · 2024-05-15T10:37:45Z

attn @AhmetNSimsek

per our discussion, here is the WIP

left to do:

refactor todo list

implement {method}_iter()
atlas elements
- space
- parcellation
  - class
  - get_region()
- region
  - class
  - find()
  - top_most flag
  - get_regional_map()
  - get_centroids an _get_spatialprops
- parcellation map @AhmetNSimsek
  - labelled
  - statistical
update siibra/attr/meta/doi -> siibra/attr/desc/url see FZJ-INM1-BDA/siibra-configurations@9806628
- update schema
- update key url -> value
update siibra/attr/data/voi -> siibra/attr/data/image see FZJ-INM1-BDA/siibra-configurations@9806628
- update schema
update regionspec see FZJ-INM1-BDA/siibra-configurations@9806628
- fix property key name -> value

port over features

connectivity
bold

live queries

allen see https://github.com/FZJ-INM1-BDA/siibra-python/blob/e2c5284f6a1f85a70556b583944339e26d960386/siibra/retrieval_new/api_fetcher/allen.py and

siibra-python/foo.py

Lines 68 to 82 in e2c5284

    
           assert siibra.descriptions.modality.vocab.GENE_EXPRESSIONS == siibra.descriptions.modality.Modality(value="Gene Expressions") 
        
           julichbrain_29_hoc1_lh_pmap = Image(format="nii", space_id="minds/core/referencespace/v1.0.0/dafcffc5-4826-4bf1-8ff6-46b8a31ff8e2", 
        
                                               fetcher="https://neuroglancer.humanbrainproject.eu/precomputed/data-repo-ng-bot/20210616-julichbrain-v2.9.0-complete-mpm/PMs/Area-hOc1/4.2/Area-hOc1_l_N10_nlin2ICBM152asym2009c_4.2_publicP_026bcbe494dc4bfe702f2b1cc927a7c1.nii.gz") 
        
           region = Region(attributes=[julichbrain_29_hoc1_lh_pmap]) 
        
           gene_maoa = Gene(value="MAOA") 
        
           gene_tac1 = Gene(value="TAC1") 
        
           query = QueryParam(attributes=[julichbrain_29_hoc1_lh_pmap, 
        
                                          gene_maoa, 
        
                                          gene_tac1, 
        
                                          siibra.descriptions.modality.vocab.GENE_EXPRESSIONS ]) 
        
           gene_expression = list(feature_get(query, Feature)) 
        
           print(gene_expression)

big brain intensity profile see d22f454
brainglobe
document how to add additional external connectors see d22f454

reimplement autocomplete all modalities see 44a7194
sample feature fetching see 44a7194

move cache to root level module
attr
- image dataitem (Image subclassing Location) @AhmetNSimsek
- add more attr comparison
  - regionspec match @AhmetNSimsek
  - Bbox to bbox see 6bb174e
- ptcloud (and pt) to image
setup & test siibra-schema repo see https://github.com/FZJ-INM1-BDA/siibra-schema
consider using standard json-ld schema, context etc
Retrieval
- Volume Fetcher @AhmetNSimsek
  - NiftiFetcher
  - NeuroglancerFetcher
  - NeuroglancerMeshFetcher
  - GiftiFetcher
  - FreesurferAnnotFetcher
- File fetcher
  - LocalFile Fetcher @xgui3783
  - zip fetcher
  - tar fetcher
  - Github fetcher @xgui3783
  - Gitlab fetcher @xgui3783
  - EBRAINS fetcher ?
  - owncloud?
Locations @AhmetNSimsek
- boundingbox
- pointcloud
  - contour
- point
- plane3d
- polyline
Map assignment
- Point and PointCloud (etc) assignment to labelled and statistical maps
- Image assignment to maps
Unit tests
e2e tests (For now, we are using foos under examples to test along with migrated/translated examples)
Examples
Notebooks to reproduce code on siibra paper
Polish docs (Need to discuss)
Docstrings

AhmetNSimsek · 2024-06-24T08:12:31Z

region.find is not deterministic and rather slow. I am noting down to investigate later.

AhmetNSimsek · 2024-06-25T06:51:47Z

@xgui3783, I think we should remove the dataclass decorators from classes that inherit other a class that has a dataclass decorator. Because it creates unexpected behaviors such as some methods are overridden and very difficult to debug as it is an unexpected behavior.
Regarding being explicit: if a class A inherits a class B with dataclass decorators, the developer would know that A is also a dataclass. So, by repeating the 2 lines required to decorate class A as a dataclass, we break DRY, we face unexpected behavior, and we don't gain anything besides being explicit which is not really necessary in this context.

This changes if the class has extra members. Then, it is needed to use the decorator as the __init__ needs to be overridden. I suggest we remove all we can and leave only the necessary implementations. What do you think?

AhmetNSimsek · 2024-07-05T15:34:11Z

siibra/retrieval_new/api_fetcher/brainglobe.py

@@ -107,13 +116,33 @@ def populate_regions(
        parent_region = _dict_id_to_region[parent_id]
        region.parent = parent_region

+def tiff_to_nii(tiff_bytes: bytes, affine: np.ndarray) -> str:


I think this should go to commons_new.maps or we create a tiff fetcher.
I prefer the second one (EDT: tho it is difficult to assume tiffs are standard. We could make tiff.brainglobe_tiff_fetcher)

as you alluded to, there is no standard tiff format.

That's why I think it should be here (i.e. brainglobe standard)

AhmetNSimsek · 2024-07-05T15:39:49Z

siibra/retrieval_new/api_fetcher/brainglobe.py


    _registered_atlas.add(atlas_name)
    logger.info(f"{atlas_name} added to siibra.")

-    return space, parcellation
+    return space, parcellation, labelled_map


Just food for discussion: since a labelled_map has to have this space and parcellation as a member, or computed property, perhaps we should just return the map.
The user can then access to parcellation and space either from mp.parcellation/space or from the registry

I don't mind either way (I am also fine not returning anything)

The only reason why I return all three is to be explicit, to show all of the items that are added

AhmetNSimsek · 2024-07-30T15:26:43Z

To profile and improve:

~~get_map is very slow. It does not speed up after first use.~~ fixed @xgui3783
[ ]

Known bugs:

[ ]

Other issues:(

~~get_parcellation does not return the newest parcellation~~ : cannot reproduce @xgui3783
region.get_boundingbox(space_spec) needs to be implemented

@xgui3783 Please edit and add/update more as we move forward to keep track of finer details

fix: point add/sub. space_id should not proxy for space_spec

…le image intersection

fix: location.ops.transform pointcloud test: fixed point coordinate type

fix: nifti dtype feat: find maps can allow None parcellation/space spec

maint: sparsemap.fetch (typing + logic rework) maint: image.py assertion block & space_id test

…nd_intersecting_regions`

…points

workflow: add debug

feat: attribute_collection now exposes categorization as a human readable str feat: searchresult indexes categorizations when building summary table

fix: preprocess region

…ity) fix: big brain profile

maint: low ram/non local as perf default feat: append all dataop for volumes feat: query cursor data table & reconfigure

doc: live queries base class

fix (e2e): use correct error class feat: add siibra error class

refactor: (sparse)map.extract_full_map

feat: contain in enum

fix: merge label nifti data operation

doc: zip fetcher description test(e2e): fix bold, fix allen, fix image

xgui3783 mentioned this pull request May 14, 2024

Experimental: feature refactor using attributes #553

Closed

AhmetNSimsek assigned dickscheid, xgui3783 and AhmetNSimsek May 14, 2024

AhmetNSimsek marked this pull request as draft June 25, 2024 07:38

AhmetNSimsek reviewed Jul 5, 2024

View reviewed changes

xgui3783 and others added 19 commits August 8, 2024 11:13

fix: point.homogenous

a5a583f

fix: point add/sub. space_id should not proxy for space_spec

carry location.transform to location.ops

c959663

point.create_gaussian_kernel

11aa1b1

update typing: location.ops.warp

335695e

image.py: read values from voxels/points, resample image, update simp…

9dcb3bb

…le image intersection

boundingbox.py: remove unnecessary import

221ba7d

pointset.py -> pointcloud.py

610fa85

fix import: locations.ops.transform

2dad37a

fix: pointcloud post init sigma, typo

a24441e

fix: location.ops.transform pointcloud test: fixed point coordinate type

e2e: interection

975ba4b

fix: nifti dtype feat: find maps can allow None parcellation/space spec

test: adapt image unit test

1a3beb0

maint: sparsemap.fetch (typing + logic rework) maint: image.py assertion block & space_id test

fix: locations.ops.warp typevar, pointcloud.__iter__

360f70a

allow access to (main) location types from main package

09ce1f1

Calculate image intersection scores, refactor Map.assign -> `Map.fi…

cef0a9d

…nd_intersecting_regions`

fix import

17a1cc9

fix: Image read_voxels and read_values_at_points

ca5af10

fix image.read_values_at_points signle point bug, fix poitcloud.from_…

442f067

…points

fix pointcloud get_item

fc4631e

Update point and pointcloud typing, allow getting ndarray

3b46984

xgui3783 and others added 29 commits October 9, 2024 15:59

fix: unit test

239f955

workflow: add debug

refactor: consolidating assignment/search

f4958df

MVP search result

7667bf4

fix: region.mapped_spaces

f0eb1a9

fix: rename remnants of dataprovider to datarecipes

34b5f76

remove query from tabular recipe, update DFAccessor to take row as well

6b3a546

depricate: feature.data

1e2a64e

use reconfigure instead of get_profile

16ecfa5

maint: clarify attribute qualifcation register

1ddb25c

feat: attribute_collection now exposes categorization as a human readable str feat: searchresult indexes categorizations when building summary table

fix: notebook 4, receptor

07cb0e9

fix: typo

182119d

fix: categorization str

ad5a4f1

fix: preprocess region

fix: notebook 5 (layerwise modified silver staining -> cell body dens…

d07eb2d

…ity) fix: big brain profile

fix: genes (notebook 5)

a2e7b87

fix axes kwarg for plotting connectivity matrices with nilearn

8094630

maint: rename siibraconf -> perfconf

198ac6f

maint: low ram/non local as perf default feat: append all dataop for volumes feat: query cursor data table & reconfigure

enh: live queries mechanism

fed3167

doc: live queries base class

test: bbox zero volume check

126dc04

fix: imports

5702abb

fix (e2e): use tmp dir for local test

b9b5eb3

fix (e2e): use correct error class feat: add siibra error class

fix(typing): mapping typed dict

babd1cd

refactor: (sparse)map.extract_full_map

maint: formalize volume format registration

b74cf21

feat: contain in enum

fix: allow map extract_regional_map to accept node region

f2909e7

feat: datarecipe.describe now returns description of each step

87bbf7a

test(e2e): fix allen gene test

3a572c0

fix: boundingbox.from_array

62f1d6b

fix: merge label nifti data operation

feat: ng allow fetch channel

f5b572c

doc: zip fetcher description test(e2e): fix bold, fix allen, fix image

test: fix coverage

1f63016

doc: update version of upload-artifact

bfa704d

AhmetNSimsek added the version 2.0.0 label Jan 14, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: use attributes #594

refactor: use attributes #594

xgui3783 commented May 14, 2024 •

edited by AhmetNSimsek

Loading

xgui3783 commented May 15, 2024 •

edited by AhmetNSimsek

Loading

AhmetNSimsek commented Jun 24, 2024

AhmetNSimsek commented Jun 25, 2024 •

edited

Loading

AhmetNSimsek Jul 5, 2024 •

edited

Loading

xgui3783 Jul 5, 2024

AhmetNSimsek Jul 5, 2024

xgui3783 Jul 5, 2024

AhmetNSimsek commented Jul 30, 2024 •

edited by xgui3783

Loading

refactor: use attributes #594

Are you sure you want to change the base?

refactor: use attributes #594

Conversation

xgui3783 commented May 14, 2024 • edited by AhmetNSimsek Loading

xgui3783 commented May 15, 2024 • edited by AhmetNSimsek Loading

AhmetNSimsek commented Jun 24, 2024

AhmetNSimsek commented Jun 25, 2024 • edited Loading

AhmetNSimsek Jul 5, 2024 • edited Loading

Choose a reason for hiding this comment

xgui3783 Jul 5, 2024

Choose a reason for hiding this comment

AhmetNSimsek Jul 5, 2024

Choose a reason for hiding this comment

xgui3783 Jul 5, 2024

Choose a reason for hiding this comment

AhmetNSimsek commented Jul 30, 2024 • edited by xgui3783 Loading

xgui3783 commented May 14, 2024 •

edited by AhmetNSimsek

Loading

xgui3783 commented May 15, 2024 •

edited by AhmetNSimsek

Loading

AhmetNSimsek commented Jun 25, 2024 •

edited

Loading

AhmetNSimsek Jul 5, 2024 •

edited

Loading

AhmetNSimsek commented Jul 30, 2024 •

edited by xgui3783

Loading