Improvement of the namespace/schema_url mapping

[manusimidt]

I stumbled across the following page which defines namespaces and schema urls for all taxonomies used by filers that submit to SEC EDGAR:
https://www.sec.gov/info/edgar/edgartaxonomies.xml

maybe the parser can use this xml file to map namespaces to xml schemas in the case it's not defined in the taxonomy extension schema.
Currently this is done by using the following mapping:
https://github.com/manusimidt/xbrl_parser/blob/34952a6a1185a81d767c591d3a42f53423b558d5/xbrl/taxonomy.py#L160-L188

[nevraj]

Hello @manusimidt thanks for creating and maintaining this library. It is very useful for us to parse SEC XBRL document. I was wondering if there is any timeline to implement namespace/schema_url mapping improvement as mentioned in this discussion? There are few new namespaces e.g. (xbrl.sec.gov/rr/2022, xbrl.sec.gov/dei/2023 etc.) are missing from the static ns_schema_map. Becuase of this few of the XBRL documents are failing to parse. If we can't implement above solution (by reading the current mapping from https://www.sec.gov/info/edgar/edgartaxonomies.xml), can we at least update the ns_schema_map so that we can continue with our parsing for this year's document?

[manusimidt]

Yes, thanks for brining this to my attention. I will add them as soon as possible!

[manusimidt]

@nevraj should be fixed in the newest version of py-xbrl. Please tell me if it still doesn't work for you!
If you still have failing iXBRL documents, please send me the link and/or the document so that I can investigate and fix the issue :)

[nevraj]

Thanks @manusimidt for your quick action. After comparing your recent commit and SEC site https://www.sec.gov/info/edgar/edgartaxonomies.xml - I can see below schema mappings are missing. Can you please add them and release the next version 2.2.7? Appreciate your help.

{'https://xbrl.org/2023/calculation-1.1': 'https://www.xbrl.org/2023/calculation-1.1.xsd',
'http://xbrl.org/2014/extensible-enumerations': 'http://www.xbrl.org/2014/extensible-enumerations.xsd',
'http://xbrl.sec.gov/rxp/2023': 'https://xbrl.sec.gov/rxp/2023/rxp-2023.xsd',
'http://xbrl.sec.gov/snj/2023': 'https://xbrl.sec.gov/snj/2023/snj-2023.xsd',
'http://xbrl.sec.gov/snj-def/2023': 'https://xbrl.sec.gov/snj/2023/snj-2023_def.xsd',
'http://xbrl.sec.gov/country-def/2023': 'https://xbrl.sec.gov/country/2023/country-2023_def.xsd',
'http://xbrl.sec.gov/oef/2023': 'https://xbrl.sec.gov/oef/2023/oef-2023.xsd',
'http://xbrl.sec.gov/oef-rr/2023': 'https://xbrl.sec.gov/oef/2023/oef-rr-2023.xsd',
'http://xbrl.sec.gov/oef-sr/2023': 'https://xbrl.sec.gov/oef/2023/oef-sr-2023.xsd',
'http://xbrl.sec.gov/oef-lab/2023': 'https://xbrl.sec.gov/oef/2023/oef-2023_lab.xsd',
'http://xbrl.sec.gov/oef-cal/2023': 'https://xbrl.sec.gov/oef/2023/oef-2023_cal.xsd',
'https://xbrl.ifrs.org/taxonomy/2023-03-23/ifrs-full': 'https://xbrl.ifrs.org/taxonomy/2023-03-23/full_ifrs/full_ifrs-cor_2023-03-23.xsd',
'http://xbrl.sec.gov/rr-sub/2023': 'https://xbrl.sec.gov/rr/2023/rr-sub-2023.xsd',
'http://xbrl.sec.gov/rr-cal/2023': 'https://xbrl.sec.gov/rr/2023/rr-2023_cal.xsd',
'http://xbrl.sec.gov/dei/2023': 'https://xbrl.sec.gov/dei/2023/dei-2023.xsd',
'http://xbrl.sec.gov/dei-sub/2023': 'https://xbrl.sec.gov/dei/2023/dei-sub-2023.xsd',
'http://xbrl.sec.gov/dei-def/2023': 'https://xbrl.sec.gov/dei/2023/dei-2023_def.xsd',
'http://xbrl.sec.gov/dei-lab/2023': 'https://xbrl.sec.gov/dei/2023/dei-2023_lab.xsd',
'http://xbrl.sec.gov/dei-pre/2023': 'https://xbrl.sec.gov/dei/2023/dei-2023_pre.xsd',
'http://xbrl.sec.gov/dei/2022q4': 'https://xbrl.sec.gov/dei/2022q4/dei-2022q4.xsd',
'http://xbrl.sec.gov/dei-sub/2022q4': 'https://xbrl.sec.gov/dei/2022q4/dei-sub-2022q4.xsd',
'http://xbrl.sec.gov/dei-def/2022q4': 'https://xbrl.sec.gov/dei/2022q4/dei-2022q4_def.xsd',
'http://xbrl.sec.gov/dei-lab/2022q4': 'https://xbrl.sec.gov/dei/2022q4/dei-2022q4_lab.xsd',
'http://xbrl.sec.gov/dei-pre/2022q4': 'https://xbrl.sec.gov/dei/2022q4/dei-2022q4_pre.xsd',
'http://xbrl.sec.gov/ecd/2023': 'https://xbrl.sec.gov/ecd/2023/ecd-2023.xsd',
'http://xbrl.sec.gov/ecd-sub/2023': 'https://xbrl.sec.gov/ecd/2023/ecd-sub-2023.xsd',
'http://xbrl.sec.gov/ecd/2022q4': 'https://xbrl.sec.gov/ecd/2022q4/ecd-2022q4.xsd',
'http://xbrl.sec.gov/ecd-sub/2022q4': 'https://xbrl.sec.gov/ecd/2022q4/ecd-sub-2022q4.xsd',
'http://fasb.org/srt-sup/2022q3': 'https://xbrl.fasb.org/srt/2022q3/srt-sup-2022q3.xsd',
'http://fasb.org/us-gaap-sup/2022q3': 'https://xbrl.fasb.org/us-gaap/2022q3/us-gaap-sup-2022q3.xsd',
'http://xbrl.sec.gov/vip/2023': 'https://xbrl.sec.gov/vip/2023/vip-2023.xsd',
'http://xbrl.sec.gov/vip-n3/2023': 'https://xbrl.sec.gov/vip/2023/vip-n3-2023.xsd',
'http://xbrl.sec.gov/vip-n4/2023': 'https://xbrl.sec.gov/vip/2023/vip-n4-2023.xsd',
'http://xbrl.sec.gov/vip-n6/2023': 'https://xbrl.sec.gov/vip/2023/vip-n6-2023.xsd',
'http://xbrl.sec.gov/country/2023': 'https://xbrl.sec.gov/country/2023/country-2023.xsd',
'http://xbrl.sec.gov/currency/2023': 'https://xbrl.sec.gov/currency/2023/currency-2023.xsd',
'http://xbrl.sec.gov/vip/2022q2': 'https://xbrl.sec.gov/vip/2022q2/vip-2022q2.xsd',
'http://xbrl.sec.gov/vip-n3/2022q2': 'https://xbrl.sec.gov/vip/2022q2/vip-n3-2022q2.xsd',
'http://xbrl.sec.gov/vip-n4/2022q2': 'https://xbrl.sec.gov/vip/2022q2/vip-n4-2022q2.xsd',
'http://xbrl.sec.gov/vip-n6/2022q2': 'https://xbrl.sec.gov/vip/2022q2/vip-n6-2022q2.xsd',
'http://xbrl.sec.gov/country/2022': 'https://xbrl.sec.gov/country/2022/country-2022.xsd',
'http://xbrl.sec.gov/currency/2022': 'https://xbrl.sec.gov/currency/2022/currency-2022.xsd',
'http://xbrl.sec.gov/stpr/2022': 'https://xbrl.sec.gov/stpr/2022/stpr-2022.xsd',
'http://xbrl.sec.gov/sic/2022': 'https://xbrl.sec.gov/sic/2022/sic-2022.xsd',
'http://xbrl.sec.gov/naics/2022': 'https://xbrl.sec.gov/naics/2022/naics-2022.xsd',
'http://xbrl.sec.gov/exch/2022': 'https://xbrl.sec.gov/exch/2022/exch-2022.xsd',
'https://xbrl.ifrs.org/taxonomy/2022-03-24/ifrs-full': 'https://xbrl.ifrs.org/taxonomy/2022-03-24/full_ifrs/full_ifrs-cor_2022-03-24.xsd',
'http://fasb.org/us-gaap/2022': 'https://xbrl.fasb.org/us-gaap/2022/elts/us-gaap-2022.xsd',
'http://fasb.org/srt/2022': 'https://xbrl.fasb.org/srt/2022/elts/srt-2022.xsd',
'http://xbrl.sec.gov/dei/2022': 'https://xbrl.sec.gov/dei/2022/dei-2022.xsd',
'http://xbrl.sec.gov/dei-lab/2022': 'https://xbrl.sec.gov/dei/2022/dei-2022_lab.xsd',
'http://xbrl.sec.gov/dei-pre/2022': 'https://xbrl.sec.gov/dei/2022/dei-2022_pre.xsd',
'http://xbrl.sec.gov/dei-def/2022': 'https://xbrl.sec.gov/dei/2022/dei-2022_def.xsd',
'http://www.esma.europa.eu/xbrl/esef/arcrole/wider-narrower': 'http://www.xbrl.org/lrr/arcrole/esma-arcrole-2018-11-21.xsd',
'http://xbrl.sec.gov/exch/2023': 'https://xbrl.sec.gov/exch/2023/exch-2023.xsd',
'http://xbrl.sec.gov/sic/2023': 'https://xbrl.sec.gov/sic/2023/sic-2023.xsd',
'http://xbrl.sec.gov/stpr/2023': 'https://xbrl.sec.gov/stpr/2023/stpr-2023.xsd',
'http://xbrl.sec.gov/naics/2023': 'https://xbrl.sec.gov/naics/2023/naics-2023.xsd',
'http://fasb.org/us-gaap/2023': 'https://xbrl.fasb.org/us-gaap/2023/elts/us-gaap-2023.xsd',
'http://fasb.org/srt/2023': 'https://xbrl.fasb.org/srt/2023/elts/srt-2023.xsd',
'http://fasb.org/us-types/2023': 'https://xbrl.fasb.org/us-gaap/2023/elts/us-types-2023.xsd',
'http://fasb.org/us-roles/2023': 'https://xbrl.fasb.org/us-gaap/2023/elts/us-roles-2023.xsd',
'http://fasb.org/srt-types/2023': 'https://xbrl.fasb.org/srt/2023/elts/srt-types-2023.xsd',
'http://fasb.org/srt-roles/2023': 'https://xbrl.fasb.org/srt/2023/elts/srt-roles-2023.xsd',
'http://www.xbrl.org/2009/role/net': 'http://www.xbrl.org/lrr/role/net-2009-12-16.xsd',
'http://www.xbrl.org/2009/role/negated': 'http://www.xbrl.org/lrr/role/negated-2009-12-16.xsd',
'http://www.xbrl.org/2009/arcrole/fact-explanatoryFact': 'http://www.xbrl.org/lrr/arcrole/factExplanatory-2009-12-16.xsd',
'http://xbrl.org/2020/extensible-enumerations-2.0': 'https://www.xbrl.org/2020/extensible-enumerations-2.0.xsd',
'http://xbrl.sec.gov/dei-sub/2022': 'https://xbrl.sec.gov/dei/2022/dei-sub-2022.xsd',
'http://fasb.org/us-types/2022': 'https://xbrl.fasb.org/us-gaap/2022/elts/us-types-2022.xsd',
'http://fasb.org/us-roles/2022': 'https://xbrl.fasb.org/us-gaap/2022/elts/us-roles-2022.xsd',
'http://fasb.org/srt-types/2022': 'https://xbrl.fasb.org/srt/2022/elts/srt-types-2022.xsd',
'http://fasb.org/srt-roles/2022': 'https://xbrl.fasb.org/srt/2022/elts/srt-roles-2022.xsd',
'http://www.xbrl.org/dtr/type/2020-01-21': 'https://www.xbrl.org/dtr/type/2020-01-21/types.xsd',
'http://xbrl.sec.gov/cef/2023': 'https://xbrl.sec.gov/cef/2023/cef-2023.xsd',
'http://xbrl.sec.gov/cef-pre/2023': 'https://xbrl.sec.gov/cef/2023/cef-2023_pre.xsd',
'http://xbrl.sec.gov/cef/2022': 'https://xbrl.sec.gov/cef/2022/cef-2022.xsd',
'http://xbrl.sec.gov/vip/2022': 'https://xbrl.sec.gov/vip/2022/vip-2022.xsd',
'http://xbrl.sec.gov/vip-n3/2022': 'https://xbrl.sec.gov/vip/2022/vip-n3-2022.xsd',
'http://xbrl.sec.gov/vip-n4/2022': 'https://xbrl.sec.gov/vip/2022/vip-n4-2022.xsd',
'http://xbrl.sec.gov/vip-n6/2022': 'https://xbrl.sec.gov/vip/2022/vip-n6-2022.xsd',
'http://xbrl.sec.gov/rr/2023': 'https://xbrl.sec.gov/rr/2023/rr-2023.xsd',
'http://xbrl.sec.gov/rr-lab/2023': 'https://xbrl.sec.gov/rr/2023/rr-2023_lab.xsd',
'http://xbrl.sec.gov/rr-pre/2023': 'https://xbrl.sec.gov/rr/2023/rr-2023_pre.xsd',
'http://xbrl.sec.gov/rr-def/2023': 'https://xbrl.sec.gov/rr/2023/rr-2023_def.xsd',
'http://xbrl.sec.gov/rr/2022': 'https://xbrl.sec.gov/rr/2022/rr-2022.xsd',
'http://xbrl.sec.gov/rr-lab/2022': 'https://xbrl.sec.gov/rr/2022/rr-2022_lab.xsd',
'http://xbrl.sec.gov/rr-pre/2022': 'https://xbrl.sec.gov/rr/2022/rr-2022_pre.xsd',
'http://xbrl.sec.gov/rr-def/2022': 'https://xbrl.sec.gov/rr/2022/rr-2022_def.xsd',
'http://www.xbrl.org/dtr/type/2022-03-31': 'https://www.xbrl.org/dtr/type/2022-03-31/types.xsd',
'http://xbrl.org/2006/xbrldi': 'http://www.xbrl.org/2006/xbrldi-2006.xsd',
'http://xbrl.org/2005/xbrldt': 'http://www.xbrl.org/2005/xbrldt-2005.xsd',
'http://www.xbrl.org/2009/role/deprecated': 'http://www.xbrl.org/lrr/role/deprecated-2009-12-16.xsd',
'http://www.xbrl.org/2009/arcrole/deprecated': 'http://www.xbrl.org/lrr/arcrole/deprecated-2009-12-16.xsd',
'http://www.xbrl.org/dtr/type/numeric': 'http://www.xbrl.org/dtr/type/numeric-2009-12-16.xsd',
'http://www.xbrl.org/dtr/type/non-numeric': 'http://www.xbrl.org/dtr/type/nonNumeric-2009-12-16.xsd',
'http://www.xbrl.org/2006/ref': 'http://www.xbrl.org/2006/ref-2006-02-27.xsd',
'http://www.xbrl.org/2004/ref': 'http://www.xbrl.org/2004/ref-2004-08-10.xsd',
'http://www.w3.org/1999/xlink': 'http://www.xbrl.org/2003/xlink-2003-12-31.xsd',
'http://www.xbrl.org/2003/XLink': 'http://www.xbrl.org/2003/xl-2003-12-31.xsd',
'http://www.xbrl.org/2003/linkbase': 'http://www.xbrl.org/2003/xbrl-linkbase-2003-12-31.xsd',
'http://www.xbrl.org/2003/instance': 'http://www.xbrl.org/2003/xbrl-instance-2003-12-31.xsd'}

[nevraj]

Hi @manusimidt just wondering if you can fix above and cut the new release this week? Just planning for our release accordingly.

[manusimidt]

Yes, will have a look at it by tomorrow 15:00 (UTC+2)

[manusimidt]

Have a look at version 2.2.8

[jakemdrew]

Hello!

I ran into a similar issue today with 'The taxonomy with namespace http://xbrl.sec.gov/stpr/2022 could not be found'.

Here was my quick solution to inject the correct mapping from: https://www.sec.gov/info/edgar/edgartaxonomies.xml

from xbrl.cache import HttpCache
from xbrl.instance import XbrlParser, XbrlInstance
import xbrl

xbrl.taxonomy.ns_schema_map['http://xbrl.sec.gov/stpr/2022'] = 'https://xbrl.sec.gov/stpr/2022/stpr-2022.xsd'

Once the ns_schema_map in taxonomy.py has been updated I was able to parse the filing as expected using parser.parse()

[jakemdrew]

For a more comprehensive solution I use this:

from xbrl.cache import HttpCache
from xbrl.instance import XbrlParser, XbrlInstance
import xbrl

# Hot fix for py-xbrl package.
# 1. Combines taxonomy_schema_maps from:
#    * xbrl.taxonomy.ns_schema_map see: https://github.com/manusimidt/py-xbrl/blob/main/xbrl/taxonomy.py
#    * the sec see: https://www.sec.gov/info/edgar/edgartaxonomies.xml
# 2. Hacks py-xbrl and updates the xbrl.taxonomy.ns_schema_map with a more comprehensive set of the lastest mappings.
def update_taxonomy_schema_map(verbose = False):
    
    # Get latest schema map from the sec
    sec_schema_map = pd.read_xml('https://www.sec.gov/info/edgar/edgartaxonomies.xml')

    # Create dictionary of {k ==  Namespace : v == Href}
    sec_schema_map = pd.Series(sec_schema_map.Href.values,index=sec_schema_map.Namespace).to_dict()

    # Print schema map counts
    if verbose:
        print('sec_schema_map:', len(sec_schema_map))
        print('ns_schema_map:', len(xbrl.taxonomy.ns_schema_map))


    # Combine schema maps, favors the last dictionary for duplicate keys!!!
    sec_schema_map = {**xbrl.taxonomy.ns_schema_map, **sec_schema_map}

    # Print schema map counts
    if verbose:
        print('combined schema_map:', len(sec_schema_map))
    
    return sec_schema_map
    
xbrl.taxonomy.ns_schema_map = update_taxonomy_schema_map()

When running today with verbose == True I get the following:

sec_schema_map: 102
ns_schema_map: 345
combined schema_map: 417
xbrl.taxonomy.ns_schema_map: 417

[manusimidt]

yes, that looks like a nice workaround for now! I want to implement similar functionality as you proposed directly into py-xbrl in the future.

[nevraj]

Thanks a lot @manusimidt and @jakemdrew . Appreciate your help.