[BUG] Can't open older lightcones now

[JasperSolt]

In pulling the latest updates to try and resolve #393 , I now get the following error when I try to load in any lightcones at all:

ValueError                                Traceback (most recent call last)
Cell In[2], line 8
      4 lc_path = "/users/jsolt/data/jsolt/21cmFAST_lightcones_centralpix_v04"
      5 lightcones = os.listdir(lc_path)
----> 8 lc = p21c.outputs.LightCone.read(lightcones[0], lc_path)
     10 lc.lightcone_redshifts

File /oscar/home/jsolt/FourierNN/myenv/lib64/python3.9/site-packages/py21cmfast/outputs.py:1161, in _HighLevelOutput.read(cls, fname, direc)
   1158 park, glbls = cls._read_inputs(fname)
   1159 boxk = cls._read_particular(fname)
-> 1161 with global_params.use(**glbls):
   1162     out = cls(**park, **boxk)
   1164 return out

File /usr/lib64/python3.9/contextlib.py:119, in _GeneratorContextManager.__enter__(self)
    117 del self.args, self.kwds, self.func
    118 try:
--> 119     return next(self.gen)
    120 except StopIteration:
    121     raise RuntimeError("generator didn't yield") from None

File /oscar/home/jsolt/FourierNN/myenv/lib64/python3.9/site-packages/py21cmfast/inputs.py:340, in GlobalParams.use(self, **kwargs)
    338 for k, val in kwargs.items():
    339     if k.upper() not in this_attr_upper:
--> 340         raise ValueError(f"{k} is not a valid parameter of global_params")
    341     key = this_attr_upper[k.upper()]
    342     prev[key] = getattr(self, key)

ValueError: AVG_BELOW_SAMPLER is not a valid parameter of global_params

[daviesje]

Unfortunately since this branch is undergoing active development, there will be some backward compatibility issues like this. We are working on moving backend global parameters to the python wrapper, and if the parameter structures are different, the package doesn't know if it can use it properly for computation.

In the vast majority of cases it will be easier and safer to regenerate the lightcone, however since I know you are performing large runs there are a few workarounds.

If all you need are the output fields, and you don't need to recompute any boxes, you can load the fields using h5py like you would any other hdf5 file. If you want the convenience of the LightCone structure, you can construct your own using the fields.

I've pasted a template below using some py21cmfast you can modify for your own purposes

def read_lc_ignoreparams(lcf):
    try:
        lightcone = p21c.LightCone.read(lcf)
    except:
        try:
            park, glbls = p21c.LightCone._read_inputs(lcf)
            boxk = p21c.LightCone._read_particular(lcf)
            #this sets globals to defaults, not consistent with the lightcone but allows us to load different versions
            lightcone = p21c.LightCone(**park, **boxk)
            logger.warning(f"lightcone {lcf} is likely from a different version and may have the wrong parameter structures")
        except:
            logger.error(f"could not read lightcone {lcf}")
            raise FileExistsError
    return lightcone

keep in mind that the parameter structures (lightcone.user_params etc...) will be incorrect, but you'll still have all the boxes. If you're careful you can also replace the parameter structures manually and re-save so you can load it the normal way.

[JasperSolt]

Thanks a ton. I'll probably just use the older version when opening those boxes for now. If I need to pull a more recent version I can manually go in and delete the keys from the hdf5.