FLCAC Metadata Requirements

[dt-woods]

What are they?

[dt-woods]

@bl-young, there's been a discussion on our end whether we need to 'bulk up' the flow metadata in the baseline's JSON-LD. Currently, I'm using fedelemflowlist v.1.2.2 to map elementary flows to get their UUIDs; however, I'm seeing that the CAS number and the FEDEFL version numbers are not carried over. The Commons seems to have version 1.0 available. Any recommendations on paths forward to meet requirements and utilize exiting tools?

[dt-woods]

I was not aware of that! Thanks, Ben. I'll give it a try.
@m-jamieson, you recall any other metadata hangups with the electricity baseline for publishing on the Commons?

[m-jamieson]

Hangup might be a strong word. I think that all the unit process metadata is there to satisfy FLCAC requirements...or at least they were satisfied at the time.

Along the topic of flows though, we do create a number of product and waste flows (generated by RCRAInfo) that don't fit within the elementary flows list. There are no metadata associated with these, and unfortunately, I don't know what flow-level metadata requirements there are in the commons.

We should also probably move the waste flows from "Elementary flows/emission/waste" to "Technosphere Flows" or make a new "Waste Flows" category. Guess I'll call out @bl-young again here - has there been a standard set somewhere on waste flows as provided by StEWI?

[bl-young]

Yes RCRA... I believe we have these from USEEIO as well. When we turned that into JSON this is how it manifests:
https://www.lcacommons.gov/lca-collaboration/US_Environmental_Protection_Agency/USEEIO_v2/datasets/Flows/Waste%20Flows

Is that the best solution? I may need to take that back and think on it in terms of the specific categories/contexts for those flows. I'm open to adjusting the USEEIO repo for those flows if that makes sense and you all have a better solution.

I do recommend using the UUID function in esupy (here] for consistent UUID generation. It would be good if the same RCRA flows always had the same UUID and contexts across repos.

I also seem to recall the existing baseline having some additional elementary flows that arent in the FEDEFL, maybe they were some water flows? Happy to set something up to talk through this in more detail and/or review what you have so far if that's useful.

I'll also add I have been working on creating a schema for Sources through esupy (USEPA/esupy#51) if thats valuable for you.

[bl-young]

@dt-woods in USEPA/LCIAformatter#103 I've been wanting to use the existing function in fedelemflowlist to write flows in lciafmt to make them more consistent. I started that with USEPA/LCIAformatter@36a37c4 and USEPA/fedelemflowlist@7ed509a. I realized that the way things are currently set up, it works best to pass the zipWriter object through to fedelemflowlist. Anyhow, I don't recall exactly how its set up in eLCI, but perhaps somewhere in _save_to_json you could leverage fedelemflowlist.write_jsonld() for the elementary flows. Let me know if you give it a shot. I'm happy to work through additional changes within fedelemflowlist

[dt-woods]

I'm pretty sure our UUID generation is the same as esupy's, but good to know that we would could (maybe even should) replace it with an import to that method. It assumes that the args we send the method are consistent.
https://github.com/KeyLogicLCA/ElectricityLCI/blob/8c5c5bb1c783b27ecef79cd4fa5e86326ac39021/electricitylci/olca_jsonld_writer.py#L1858

[dt-woods]

Here's my proposed solution, which happens within olca_jsonld_writer.py:

def _flow(dict_d, flowprop, dict_s):
    """Generate a reference to a flow object.

    Called by :func:`_exchange`.

    Parameters
    ----------
    dict_d : dict
        Flow data dictionary.
    flowprop : olca_schema.FlowProperty or olca_schema.Ref
        FlowProperty or Ref to a FlowProperty object.
    dict_s : dict
        Dictionary with olca_schema root entities.

    Returns
    -------
    tuple
        olca_schema.Ref : Reference to a flow object.
        dict : Updated dictionary with olca_schema root entities.
    """
    if not isinstance(dict_d, dict):
        logging.warning("No flow data received!")
        return (None, dict_s)

    uid = _val(dict_d, 'id', '@id')
    name = _val(dict_d, 'name')
    category_path = _val(dict_d, 'category', default='')
    is_waste = "waste" in category_path.lower()

    # HOTFIX: remove technosphere/3rd party flow check;
    # it duplicates every waste flow in the JSON-LD [2023-12-05; TWD]

    # Check for flow existence
    if uid in dict_s['Flow']['ids']:
        idx = dict_s['Flow']['ids'].index(uid)
        flow = dict_s['Flow']['objs'][idx]
        logging.debug("Found previous flow, '%s'" % flow.name)
    else:
        logging.debug("Creating new flow for, '%s' (%s)" % (name, uid))
        if _uid_is_valid(uid, 3) or _uid_is_valid(uid, 4):
            # Keep the good UUID
            pass
        else:
            # Generate new v3 ID based on standard format
            logging.debug("Generating new UUID for flow, '%s'" % name)
            uid = _uid(o.ModelType.FLOW, category_path, name)

        # Correct the default flow type for waste flows.
        def_type = "ELEMENTARY_FLOW"
        if is_waste:
            dict_d['flowType'] = "WASTE_FLOW"
            def_type = "WASTE_FLOW"

        f_type = _flow_type(_val(dict_d, 'flowType', default=def_type))
        flow = o.new_flow(name, f_type, flowprop)

        # Get FEDEFL version number and CAS number
        fed_version = _get_fedefl_version()
        fed_cas, fed_form, fed_syn = _get_fedefl_meta(uid)

        # Hotfix: Add missing flow metadata.
        # https://github.com/USEPA/ElectricityLCI/discussions/239
        flow.id = uid
        flow.category = category_path
        flow.description = "Based on FEDELEMFLOW version %s" % fed_version
        flow.cas = fed_cas
        flow.formula = fed_form
        flow.synonyms = fed_syn

        # Update master list
        dict_s['Flow']['ids'].append(uid)
        dict_s['Flow']['objs'].append(flow)
    return (flow.to_ref(), dict_s)


def _get_fedefl_meta(flow_id):
    """Query the FEDEFL for flow metadata (i.e., CAS No, formula, and synonyms)

    Parameters
    ----------
    flow_id : str
        Flow universally unique identifier (mapped to FEDEFL).

    Returns
    -------
    tuple
        A tuple of length three:

        - str: CAS No
        - str: Formula
        - str: Synonyms
    """
    # Initialize return values
    f_cas = ""
    f_form = ""
    f_syn = ""

    # Read the FEDEFL into memory
    flowlist = fedelemflowlist.get_flows(False)

    # Query the flow ID and count rows that match
    tmp_df = flowlist.query("`Flow UUID` == '%s'" % flow_id)
    num_matches = len(tmp_df)

    # Query results for metadata.
    if num_matches == 0:
        logging.info("Failed to find FEDEFL flow (%s)!" % flow_id)
    elif num_matches == 1:
        # Convert data frame to series
        tmp_df = tmp_df.iloc[0]
        f_cas = tmp_df['CAS No']
        f_form = tmp_df['Formula']
        f_syn = tmp_df['Synonyms']
    else:
        logging.warning(
            "Found %d matches for FEDEFL flow (%s); using first instance." % (
                num_matches, flow_id)
        )
        tmp_df = tmp_df.iloc[0]
        f_cas = tmp_df['CAS No']
        f_form = tmp_df['Formula']
        f_syn = tmp_df['Synonyms']

    return (f_cas, f_form, f_syn)


def _get_fedefl_version():
    """Return the version string of Federal Elementary Flow List

    Currently hard-coded to return version 1.0.0 as defined in USEPA's
    LCIAfmt. See https://github.com/USEPA/fedelemflowlist/issues/199.

    Returns
    -------
    str
        Version string (e.g., '1.0.0' or 'n/a' if not available)
    """
    try:
        # Wishy thinking
        return fedelemflowlist.__version__
    except AttributeError:
        return "1.0.0"

[bl-young]

for _get_fedefl_version(), (pending USEPA/fedelemflowlist#199) could you instead use:

fedelemflowlist.globals.flow_list_specs['list_version']

Though really, I'll caution that there is a distinction between the version of the package and the version of the flowlist (you could have the latest version of fedelemflowlist but you might have an older version of the FlowList data file (parquet) stored locally which could get used).

[dt-woods]

Good to know! Yeah, I want the version of the flow list, not the Python package. I'll incorporate the code you sent. Thanks, Ben!

[dt-woods]

What I'm seeing as a means to getting the file version is the following:

from esupy.processed_data_mgmt import find_file
import fedelemflowlist

def _get_fedefl_version():
    """Return the version string of local Federal Elementary Flow List

    Returns
    -------
    str
        Version string. Defaults to '1.0.0' if methods fail.
    """
    version = None
    try:
        # Adapted from:
        # https://github.com/USEPA/ElectricityLCI/discussions/239
        meta = fedelemflowlist.globals.set_metadata()
        fpath = fedelemflowlist.globals.fedefl_path

        # From esupy's documentation: returns the most recent version of the
        # file requested---so does not have to match ``meta.tool_version``.
        f = find_file(meta, fpath)
        if f:
            # The parquet file name is based on ``set_metadata`` function 
            # in fedelemflowlist's globals.py.
            r = re.compile(".*_v(\d+\.\d+\.\d+)_.*")
            m = r.match(f.name)
            if m:
                version = m.group(1)
        if version is None:
            version = fedelemflowlist.globals.flow_list_specs['list_version']
    except AttributeError:
        version = "1.0.0"
    
    return version

[bl-young]

Moving forward we will be distinguishing the package version from the list version (see USEPA/fedelemflowlist#205). Did you end up implementing this function above as I think it could be useful to move into fedelemflowlist itself.

[dt-woods]

Running flowlist = fedelemflowlist.get_flows(False) for each flow is laughably slow and litters the info log with thousands of processed_data_mgmt:load_preprocessed_output:Returning ~/fedelemflowlist/FedElemFlowListMaster_v1.2.0_e57a542.parquet

[bl-young]

see my proposed solution in KeyLogicLCA#12

[dt-woods]

Despite the slowness, it gets the job done.

[dt-woods]

I think it would be a benefit to move this step to the post processing phase.

[dt-woods]

How do I know what version of the parquet file I have?

…

On Thu, Aug 29, 2024, 14:12 Ben Young (ERG) ***@***.***> wrote: for _get_fedefl_version(), could you instead use: fedelemflowlist.globals.flow_list_specs['list_version'] Though really, I'll caution that there is a distinction between the version of the package and the version of the flowlist (you could have the latest version of fedelemflowlist but you might have an older version of the FlowList data file (parquet) stored locally which could get used). — Reply to this email directly, view it on GitHub <#239 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHCFB5MHPOX5QAUBBYQZWF3ZT5P7XAVCNFSM6AAAAABGTK5GK6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANBZGA2DGNI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[bl-young]

there should be a metadata file (json) that is downloaded as well that indicates that. Also the name of the file should include the version number.