Method for estimating ambient temperature from measured GHI and clear sky conditions

[mikofski]

Is your feature request related to a problem? Please describe.
In my experience, the SURFRAD datasets were missing a lot of air temperature measurements so I needed a way to estimate ambient air temperature when working SURFRAD data on https://github.com/mikofski/PVRW2021

Describe the solution you'd like
What I did was to scale the clear sky air temps from rdtools using the ratio of total daily measured GHI to total daily clear sky GHI according to this formula:

which is expressed in this function:estimate_air_temp

def estimate_air_temp(year_start, surfrad, lat, lon, cs):
    """
    Use clear sky temps scaled by daily ratio of measured to clear sky global
    insolation.

    Parameters
    ----------
    year_start : str
        SURFRAD data year
    surfrad : pandas.DateFrame
        surfrad data frame
    lat : float
        latitude in degrees north of equator [deg]
    lon : float
        longitude in degrees east of prime meridian [deg]
    cs : pandas.DataFrame
        clear sky irradiances [W/m^2]

    Returns
    -------
    est_air_temp : pandas.DataFrame
        estimated air temperature in Celsius [C]
    temp_adj : pandas.Series
        temperature adjustment [C}
    ghi_ratio : pandas.Series
        ratio of  daily SURFRAD to clearsky GHI insolation
    daily_delta_temp : numpy.array
        daily temperature range, max - min, in Kelvin [K]
    cs_temp_air : pandas.Series
        clear sky air temperatures in Celsius [C]

    """
    daze = 367 if calendar.isleap(int(year_start)) else 366
    # create a leap year of minutes for the given year at UTC
    year_minutes = pd.date_range(
        start=year_start, freq='T', periods=daze*DAYMINUTES, tz='UTC')
    # clear sky temperature
    cs_temp_air = rdtools.clearsky_temperature.get_clearsky_tamb(
        year_minutes, lat, lon)
    # organize by day
    cs_temp_daily = cs_temp_air.values.reshape((daze, DAYMINUTES)) + KELVINS
    # get daily temperature range
    daily_delta_temp = np.array([td.max()-td.min() for td in cs_temp_daily])
    daily_delta_temp = pd.Series(
        daily_delta_temp, index=cs_temp_air.resample('D').mean().index)
    # calculate ratio of daily insolation versus clearsky
    ghi_ratio = surfrad.ghi.resample('D').sum() / cs.ghi.resample('D').sum()
    ghi_ratio = ghi_ratio.rename('ghi_ratio')
    # apply ghi ratio to next day, wrap days to start at day 1
    day1 = ghi_ratio.index[0]
    ghi_ratio.index = ghi_ratio.index + to_offset('1D')
    # set day 1 estimated air temp equal to last day
    ghi_ratio[day1] = ghi_ratio.iloc[-1]
    # fix day 1 is added last, so out of order
    ghi_ratio = ghi_ratio.sort_index()
    # scale daily temperature delta by the ratio of insolation from day before
    temp_adj = (ghi_ratio - 1.0)*daily_delta_temp[ghi_ratio.index]  # use next day
    temp_adj = temp_adj.rename('temp_adj')
    # interpolate smoothly, but fill forward minutes in last day
    est_air_temp = pd.concat(
        [cs_temp_air,
         ghi_ratio.resample('1min').interpolate(),
         temp_adj.resample('1min').interpolate()], axis=1).pad()
    # Tadj = Tcs + (GHI/CS_GHI - 1) * DeltaT 
    # if GHI/CS_GHI > 1 then adjustment > DeltaT
    est_air_temp['Adjusted Temp (C)'] = (
        est_air_temp['Clear Sky Temperature (C)'] + est_air_temp.temp_adj)
    return est_air_temp, temp_adj, ghi_ratio, daily_delta_temp, cs_temp_air

Describe alternatives you've considered

• back fill the missing air temp data in the SURFRAD datasets
• Check in ERA or MERRA or somewhere else
• just use the ambient temperature in the SURFRAD data and don't worry about missing data

Additional context
I've already implemented this in the PVRW2021 repo above, and if you look near the bottom of this notebook: multiyear_data.ipynb you can see what the temperatures look like compared to GHI and clear sky conditions:

for 1st week in January:
blue = Tclearsky, green = Tadj, red = clear sky GHI, orange = GHI

for last week in December:
blue = Tclearsky, green = Tadj, red = clear sky GHI, orange = GHI

Sorry for lack of legend and axes, they're there but I can't change the transparent background. To me this looks like it's working as expected. If the day has higher GHI then clear sky, then the ambient air temp is higher than clear sky, and v. v. Also the temperature curve is smooth.

[kandersolar]

Or maybe the ambient temperature is already in the SURFRAD data and I just don't realize it?

I think it is, you just need to know which column it's in... here's a quick comparison between the raw data and a plot from their website, not sure what the deal is with the missing values but at least it's in there:

import pandas as pd
import matplotlib.pyplot as plt
# quick and dirty
df = pd.read_fwf(r'https://gml.noaa.gov/aftp/data/radiation/surfrad/Desert_Rock_NV/2021/dra21001.dat')
tamb = df.iloc[:, -10] * 9/5 + 32
plt.plot(tamb)

Red plot is from here: https://gml.noaa.gov/grad/surfrad/metplot.html, it's Jan 1 2021 Desert Rock. And here's the readme where I found that temp is the fifth column from the right, x2 for the QC cols so it's column -10. https://gml.noaa.gov/aftp/data/radiation/surfrad/Desert_Rock_NV/README

P.S. Bill Marion put nicer (SAM CSV format, IIRC) SURFRAD data files on the DuraMAT DataHub, although they're frozen so they don't have the recent data. https://datahub.duramat.org/dataset/sioux-falls-south-dakota-usa

[mikofski]

Ok, what was I thinking then? Air temp is already parsed in pvlib.iotools.read_surfrad so why didn't I use it? Could it be that the data had issues? There were a few columns that I didn't trust, maybe air temp was one of them?

[mikofski]

In cell 13 I thought this:

# we don't trust SURFRAD DNI or DHI b/c it's often missing
# dni = df.dni.values
# dhi = df.dhi.values

# we also don't trust air temp, RH, or pressure, same reason
# temp_air = df.temp_air.values
# relative_humidity = df.relative_humidity.values
# pressure = df.pressues.values
# wind_speed = df.wind_speed.values

So I knew it was there, I just didn't like it

[mikofski]

I updated the issue description to restate that yes, the air temp data is in SURFRAD, but there are many missing timestamps, so that was my motivation. But I'm thinking this could be useful for other cases where only irradiance is available.