Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate step 2 #44

Merged
merged 17 commits into from
Dec 15, 2023
Merged

Integrate step 2 #44

Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
f0ef135
finished step2 integration
kmilo9999 Dec 6, 2023
6c852f5
Merge branch 'main' into feat-step2-intg
kmilo9999 Dec 6, 2023
e71e1c7
added missed file B02_make_spectra_gFT. modifed workflow to test step2
kmilo9999 Dec 6, 2023
8820d2d
add lmfit module to pyproject.toml
kmilo9999 Dec 6, 2023
e27ee64
removing sttep 2 from test workflow
kmilo9999 Dec 6, 2023
3dac298
removing dependency to test workkflow
kmilo9999 Dec 6, 2023
28b253d
adding lmfit again to test step 1
kmilo9999 Dec 7, 2023
ce05d64
adding step 2 in workflow
kmilo9999 Dec 7, 2023
0f706f0
add pytables as dependency
kmilo9999 Dec 7, 2023
701d18d
remove pytables as dependency
kmilo9999 Dec 7, 2023
ae9fba6
added tables depedency
kmilo9999 Dec 7, 2023
0a4edaa
removing B02_make_spectra_gFT.py file from root folder
kmilo9999 Dec 11, 2023
b02a4b2
renaming step workflow and removing unnecessary code
kmilo9999 Dec 12, 2023
e3ebaf8
new sliderule version came out a few weeks ago and the pipeline seems…
kmilo9999 Dec 13, 2023
7e93e0b
Apply suggestions from code review
kmilo9999 Dec 14, 2023
8d210c7
Apply suggestions from code review
kmilo9999 Dec 15, 2023
67846f0
removed all commented code and formating the file B02_make_spectra_gF…
kmilo9999 Dec 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/test-B01_SL_load_single_file.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,3 +23,5 @@ jobs:
run: pip install .
- name: first step B01_SL_load_single_file
run: python src/icesat2_tracks/analysis_db/B01_SL_load_single_file.py 20190502052058_05180312_005_01 SH_testSLsinglefile2 True
- name: second step B02_make_spectra_gFT
run: python src/icesat2_tracks/analysis_db/B02_make_spectra_gFT.py SH_20190502_05180312 SH_testSLsinglefile2 True
6 changes: 4 additions & 2 deletions pyproject.toml
View file
Open in desktop
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't seem to be able to install this using the description in the README any more – I'm getting the following error:

  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [12 lines of output]
      /var/folders/n5/6b48sz2j3yldl4mnglvsr6mh0000gq/T/H5closem9scgpzm.c:1:10: fatal error: 'H5public.h' file not found
      #include "H5public.h"
               ^~~~~~~~~~~~
      1 error generated.
      cpuinfo failed, assuming no CPU features: 'flags'
      * Using Python 3.9.18 (main, Oct  4 2023, 15:25:11)
      * Found cython 3.0.6
      * USE_PKGCONFIG: True
      .. ERROR:: Could not find a local HDF5 installation.
         You may need to explicitly state where your local HDF5 headers and
         library can be found by setting the ``HDF5_DIR`` environment
         variable or by using the ``--hdf5`` command-line option.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Is there something special I need to do to make this work? On main it still builds fine. It looks like a problem with tables.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kmilo9999 check when tables is being used.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hollandjg This error is because it's not finding HDF5 in your macos. I created a workflow that checks the installation in that system and they seem to work. Have you set the HDF5_DIR environmental variable?

@cpaniaguam @hollandjg This is the stack trace of the throw error when the pytables dependency is missed.

Traceback (most recent call last):
  File "/mnt/c/Projects/Github/icesat2-tracks/src/icesat2_tracks/analysis_db/B02_make_spectra_gFT.py", line 439, in <module>
    MT.save_pandas_table(Pars_optm, save_name+'_params', save_path )
  File "/mnt/c/Projects/Github/icesat2-tracks/src/icesat2_tracks/local_modules/m_tools_ph3.py", line 282, in save_pandas_table
    with HDFStore(save_path+'/'+ID+'.h5') as store:
  File "/mnt/c/Projects/Github/icesat2-tracks/venv39/lib/python3.9/site-packages/pandas/io/pytables.py", line 566, in __init__
    tables = import_optional_dependency("tables")
  File "/mnt/c/Projects/Github/icesat2-tracks/venv39/lib/python3.9/site-packages/pandas/compat/_optional.py", line 135, in import_optional_dependency
    raise ImportError(msg)
ImportError: Missing optional dependency 'pytables'.  Use pip or conda to install pytables.

Basically, when the analysis results are being save to a pandas table, pandas calls the function :
import_optional_dependency("tables")
and that producess the error

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This solves it, but I think it needs to be in the README. I've put in a suggestion in #49 which can be merged into this branch.

Original file line number Diff line number Diff line change
Expand Up @@ -111,8 +111,10 @@ dependencies = [ # Optional
"mpi4py",
"icesat2-toolkit==1.0.0.22",
"geopandas",
"sliderule",
"ipyleaflet"
"sliderule==4.1.0",
"ipyleaflet",
"lmfit",
"tables"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might be able to remove this dep if the I/O is optimized in step 2.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be nice.

]

# List additional groups of dependencies here (e.g. development
Expand Down
7 changes: 4 additions & 3 deletions src/icesat2_tracks/ICEsat2_SI_tools/generalized_FT.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@

import numpy as np

import spectral_estimates as spec
import icesat2_tracks.ICEsat2_SI_tools.spectral_estimates as spec
import icesat2_tracks.ICEsat2_SI_tools.lanczos as lanczos

def rebin(data, dk, return_edges =False):
"""
Expand All @@ -25,7 +26,7 @@ def smooth_data_to_weight(dd, m=150):
dd is the data
m is the number of points to smooth over
"""
import lanczos

dd_fake = np.ones( 4*m + dd.size)*dd.max()*0.01
dd_fake[2*m:-2*m]=dd

Expand Down Expand Up @@ -916,7 +917,7 @@ def model_func(self, f, params):
return self.non_dim_spec_model(f, params['f_max'].value, params['amp'].value, params['gamma'].value)

def non_dim_spec_model(self, f, f_max, amp, gamma=1, angle_rad = 0):
import JONSWAP_gamma as spectal_models
import icesat2_tracks.local_modules.JONSWAP_gamma as spectal_models
U= 20 # results are incensitive to U
f_true = f * np.cos(angle_rad)
model= spectal_models.JONSWAP_default_alt(f_true, f_max, 20 , gamma=gamma)
Expand Down
84 changes: 29 additions & 55 deletions analysis_db/B02_make_spectra_gFT.py → ...racks/analysis_db/B02_make_spectra_gFT.py
View file
Open in desktop
kmilo9999 marked this conversation as resolved.
Show resolved Hide resolved
kmilo9999 marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean as a whistle!

Original file line number Diff line number Diff line change
@@ -1,35 +1,41 @@
# %%

import os, sys
#execfile(os.environ['PYTHONSTARTUP'])


"""
This file open a ICEsat2 track applied filters and corections and returns smoothed photon heights on a regular grid in an .nc file.
This is python 3
"""

exec(open(os.environ['PYTHONSTARTUP']).read())
exec(open(STARTUP_2021_IceSAT2).read())
from icesat2_tracks.config.IceSAT2_startup import (
mconfig,
xr,
color_schemes,
font_for_pres,
plt,
np
)


#%matplotlib inline
from threadpoolctl import threadpool_info, threadpool_limits
from pprint import pprint


import ICEsat2_SI_tools.convert_GPS_time as cGPS
import icesat2_tracks.ICEsat2_SI_tools.convert_GPS_time as cGPS
import h5py
import ICEsat2_SI_tools.io as io
import ICEsat2_SI_tools.spectral_estimates as spec
import icesat2_tracks.ICEsat2_SI_tools.io as io
import icesat2_tracks.ICEsat2_SI_tools.spectral_estimates as spec

import time
import imp
import copy
import spicke_remover
from icesat2_tracks.local_modules.m_spectrum_ph3 import spicke_remover
import datetime
import generalized_FT as gFT
import icesat2_tracks.ICEsat2_SI_tools.generalized_FT as gFT
from scipy.ndimage.measurements import label
import icesat2_tracks.local_modules.m_tools_ph3 as MT
from icesat2_tracks.local_modules import m_general_ph3 as M

# from guppy import hpy
# h=hpy()
# h.heap()
#import s3fs
#from memory_profiler import profile
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not delete these too?

Expand All @@ -50,13 +56,8 @@ def linear_gap_fill(F, key_lead, key_int):
return y_g


# %%

track_name, batch_key, test_flag = io.init_from_input(sys.argv) # loads standard experiment
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might change after the cli is implemented.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Absolutely!

#track_name, batch_key, test_flag = '20190605061807_10380310_004_01', 'SH_batch01', False
#track_name, batch_key, test_flag = '20190601094826_09790312_004_01', 'SH_batch01', False
#track_name, batch_key, test_flag = '20190207111114_06260210_004_01', 'SH_batch02', False
#track_name, batch_key, test_flag = '20190208152826_06440210_004_01', 'SH_batch01', False
#track_name, batch_key, test_flag = '20190213133330_07190212_004_01', 'SH_batch02', False
#track_name, batch_key, test_flag = '20190205231558_06030212_004_01', 'SH_batch02', False
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might have missed suggesting removing other commented code, but lines 61-76, 78, 106 could be deleted.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed all the commented code


#local track
Expand Down Expand Up @@ -87,7 +88,7 @@ def linear_gap_fill(F, key_lead, key_int):
MT.mkdirs_r(plot_path)
MT.mkdirs_r(save_path)
bad_track_path =mconfig['paths']['work'] +'bad_tracks/'+ batch_key+'/'
# %%


all_beams = mconfig['beams']['all_beams']
high_beams = mconfig['beams']['high_beams']
Expand All @@ -104,7 +105,7 @@ def linear_gap_fill(F, key_lead, key_int):
Gd = h5py.File(load_path +'/'+track_name + '_B01_binned.h5', 'r')
#Gd.close()

# %% test amount of nans in the data
# test amount of nans in the data

nan_fraction= list()
for k in all_beams:
Expand All @@ -129,7 +130,7 @@ def linear_gap_fill(F, key_lead, key_int):
print('exit.')
exit()

# %% test LS with an even grid where missing values are set to 0
# test LS with an even grid where missing values are set to 0
imp.reload(spec)
print(Gd.keys())
Gi =Gd[ list(Gd.keys())[0] ] # to select a test beam
Expand Down Expand Up @@ -161,35 +162,9 @@ def linear_gap_fill(F, key_lead, key_int):
dk = 2 * np.pi/ dlambda
kk = np.arange(0, 1/lambda_min, 1/dlambda) * 2*np.pi
kk = kk[k_0<=kk]
#dk = np.diff(kk).mean()
print('2 M = ', kk.size *2 )


# for k in all_beams:
# #I = G_gFT[k]
# I2 = Gd_cut
# #plt.plot(I['x_coord'], I['y_coord'], linewidth =0.3)
# plt.plot( I2['x']/1e3, I2['dist']/1e3)

print('2 M = ', kk.size *2 )

# # %%
# xscale= 1e3
# F= M.figure_axis_xy(5, 3, view_scale= 0.6)
# for k in all_beams:
# I = Gd[k]#['x']
# #I = Gd_cut
# plt.plot( I['x'][:]/xscale , I['y'][:]/xscale , '.' , markersize = 0.3)
# #plt.xlim(3e6, 3.25e6)
#
# #F.ax.axhline(0, color='gray', zorder= 2)
#
# plt.title('B01 filter and regrid | ' + track_name +'\npoleward '+str(track_poleward)+' \n \n', loc='left')
# plt.xlabel('along track distance (km)')
# plt.ylabel('across track distance (km)')

# %%

#Gd.keys()
print('define global xlims')
dist_list = np.array([np.nan, np.nan])
for k in all_beams:
Expand Down Expand Up @@ -226,7 +201,7 @@ def linear_gap_fill(F, key_lead, key_int):
print('set xlims: ', xlims)
print('Loop start: ', tracemalloc.get_traced_memory()[0]/1e6, tracemalloc.get_traced_memory()[1]/1e6)

# %%

G_gFT= dict()
G_gFT_x = dict()
G_rar_fft= dict()
Expand Down Expand Up @@ -263,7 +238,7 @@ def linear_gap_fill(F, key_lead, key_int):

# compute slope spectra !!
dd = np.gradient(dd)
dd, _ = spicke_remover.spicke_remover(dd, spreed=10, verbose=False)
dd, _ = spicke_remover(dd, spreed=10, verbose=False)
dd_nans = (np.isnan(dd) ) + (Gd_cut['N_photos'] <= 5)

# dd_filled = np.copy(dd)
Expand Down Expand Up @@ -302,7 +277,7 @@ def linear_gap_fill(F, key_lead, key_int):

xi_1=GG_x.x[i]
xi_2=GG_x.x[i+1]
#if k%2 ==0:


F = M.figure_axis_xy(16, 2)
eta = GG_x.eta
Expand Down Expand Up @@ -453,10 +428,10 @@ def get_stancil_nans(stancil):
del Gd_cut
Gd.close()

# %% save fitting parameters
# save fitting parameters
MT.save_pandas_table(Pars_optm, save_name+'_params', save_path )

# %% repack data
# repack data
def repack_attributes(DD):
#DD = G_LS
attr_dim_list = list(DD.keys())
Expand Down Expand Up @@ -505,7 +480,7 @@ def make_dummy_beam(GG, beam):
G_rar_fft = repack_attributes(G_rar_fft)


# %% save results
# save results
G_gFT_DS = xr.merge(G_gFT.values())#, compat='override')
G_gFT_DS['Z_hat_imag'] = G_gFT_DS.Z_hat.imag
G_gFT_DS['Z_hat_real'] = G_gFT_DS.Z_hat.real
Expand All @@ -524,4 +499,3 @@ def make_dummy_beam(GG, beam):

print('saved and done')

# %%
2 changes: 2 additions & 0 deletions src/icesat2_tracks/local_modules/m_spectrum_ph3.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
from scipy.special import gammainc
from scipy import signal
import matplotlib.pyplot as plt
from icesat2_tracks.local_modules import m_general_ph3 as M

try:
import mkl
np.use_fastnumpy = True
Expand Down