Skip to content

Commit

Permalink
Efficient way to convert a structured numpy array into a 3d array.
Browse files Browse the repository at this point in the history
This CL uses a more efficient way to convert the numpy structured array removing the need to copy the array into memory as a list. Using `.view()` doesn't seem to change the data buffer and should be more memory efficient.

closes #9

PiperOrigin-RevId: 587594128
  • Loading branch information
Xee authors committed Dec 27, 2023
1 parent 7bb1407 commit 57d7a6d
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 4 deletions.
13 changes: 9 additions & 4 deletions xee/ext.py
Original file line number Diff line number Diff line change
Expand Up @@ -464,10 +464,15 @@ def image_to_array(
pixels_getter, params, catch=ee.ee_exception.EEException
)

# TODO(#9): Find a way to make this more efficient. This is needed because
# `raw` is a structured array of all the same dtype (i.e. number of images).
arr = np.array(raw.tolist(), dtype=dtype)
data = arr.T
x_size = kwargs['grid']['dimensions']['width']
y_size = kwargs['grid']['dimensions']['height']

# This is needed because `raw` is a structured array of all the same dtype
# (i.e. number of images).
arr = raw.view(dtype).reshape(-1, x_size, y_size,)
arr = np.moveaxis(arr, 0, -1)

data = arr
current_mask_value = np.array(self.mask_value, dtype=data.dtype)
# Sets EE nodata masked value to NaNs.
data = np.where(data == current_mask_value, np.nan, data)
Expand Down
1 change: 1 addition & 0 deletions xee/ext_integration_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ def setUp(self):
)

def test_creates_lat_long_array(self):
init_ee_for_tests()
arr = xee.EarthEngineBackendArray('longitude', self.lnglat_store)
self.assertEqual((1, 360, 180), arr.shape)

Expand Down

0 comments on commit 57d7a6d

Please sign in to comment.