How is the order of facts for the instance determined?

[stkerr]

Hi! Really enjoying py-xbrl so far!

I am trying to separate out which facts correspond to which presentation linkbases in the instance and having some trouble. My first approach was to search for a fact (e.g. "Revenue") and use the first one found. This works fine for some cases but not most. I also tried to find all facts for a given presentation linkbase, look at which contexts had the most facts in common and use that. This got closer, but still doesn't work in enough cases.

Then I thought that the fact list may be sorted by the order it was consumed. So I tried iterating over each presentation linkbase along with the list of facts and "chomping" the front of the fact list as the presentation dictionary called for various facts. However, it appears that the facts are not strictly ordered by how the presentation dictionary uses them.

Do you have any guidance on how the facts are ordered in the instance or how I can determine which facts are used for a given presentation linkbase?

[manusimidt]

Hey, happy to hear its useful to you!

The order of the facts in the instance.facts list should correspond to the original order of the facts in the instance document. However, I am not entirely sure about this because normally the order of the facts shouldn't matter (and thus I did not test this previously).

And yes, it is to be expected that the order of the facts in the presentation linkbase does not match the order of the facts in the instance document. It can be that a concept is included in multiple different parts of the presentation linkbase but only defined once in the XBRL Instance document.

If I understand you correctly, I would tackle it this way:
Parse the presentation linkbase first, then loop over every entry in the presentation linkbase and try to find a fact tagged with this concept in the instance document.

However, keep in mind, that you have to deal with the dimensions that might be attached to a fact. For example If you want to search for the fact tagged with "Revenue", you might find one fact tagged with only "Revenue" and one tagged with "Revenue" and a dimension "Business Segment: Household Appliances". If you want to create a company-wide financial statement, you should only take the fact that has no dimension.

Just for the understanding: Here a quick JSON representation of a US-GAAP presentation linkbase

[pnatusch]

While I'm an experienced programmer, I'm new to XBRL and I've been struggling trying to get a handle on it. I've read XBRL for dummies and The XBRL book by Ghislain Fourny. Based on Chapter 5 (Presentation) in "The XBRL book", I got the understanding that from the presentation linkbase that I could get the correct ordering and then get the facts from the instance. Does that make sense?

As my first "project", I'm just trying to parse the xbrl documents and print out a balance sheet in the correct order.

Randomly, I chose to parse the IBM presentation linkbase, however, after parsing, the linkbase object was empty. What am I doing wrong? If this is a bug, I'm happy to help debug, but my assumption is that I'm just doing something stupid, so I didn't want to go down that path

linkbase_url: str = 'https://www.sec.gov/Archives/edgar/data/51143/000155837023002376/ibm-20221231_pre.xml'
linkbase: Linkbase = parse_linkbase_url(linkbase_url, LinkbaseType.LABEL, cache)
(Pdb) linkbase.type
<LinkbaseType.LABEL: 4>
(Pdb) linkbase.linkbase_uri
'https://www.sec.gov/Archives/edgar/data/51143/000155837023002376/ibm-20221231_pre.xml'
(Pdb) linkbase.type
<LinkbaseType.LABEL: 4>
(Pdb) linkbase.linkbase_uri
'https://www.sec.gov/Archives/edgar/data/51143/000155837023002376/ibm-20221231_pre.xml'
(Pdb) linkbase.to_dict()
{'standardExtendedLinkElements': []}
(Pdb) linkbase.to_simple_dict()
{'standardExtendedLinkElements': []}
(Pdb) linkbase.extended_links
[]
(Pdb)

First couple lines from presentation linkbase file

<link:linkbase xsi:schemaLocation="http://www.xbrl.org/2003/instance http://www.xbrl.org/2003/xbrl-instance-2003-12-31.xsd">


<roleRef xlink:type="simple" xlink:href="ibm-20221231.xsd#StatementConsolidatedStatementOfComprehensiveIncome" roleURI="http://www.ibm.com

[pnatusch]

I decided to play a little more and I realized that as I expected I was being stupid. I still have work to do but I've made some progress

linkbase_url: str = 'https://www.sec.gov/Archives/edgar/data/51143/000155837023002376/ibm-20221231_pre.xml'

I blindly copied from one of the tests, I didn't change the LinkbaseType

#WRONG: linkbase: Linkbase = parse_linkbase_url(linkbase_url, LinkbaseType.LABEL, cache)
linkbase: Linkbase = parse_linkbase_url(linkbase_url, LinkbaseType.PRESENTATION, cache)

(Pdb) linkbase.type
<LinkbaseType.LABEL: 4>
There is info in the dictionaries. I need to leave now, but it appears that I have some data to work with

[stkerr]

Thanks for the reply! That helps.

I updated my code & it can handle dimensions on the same fact reasonably well - it can differentiate Revenue (Total) from Revenue (Hardware/Software/etc). What I'm stuck at with that approach though is that the same fact name may be present multiple times & used in different parts of the statement, such as in the cash flow statement and then later in a footnote. I'm not sure how to differentiate which version to use, which means that I will be returning the wrong one & looking at a context that is for a footnote-only and not a cash flow statement.

For a more concrete, consider this 10-Q from AMD. There are 26 results for the fact net income (loss) attributable to parent. 4 of them are used in the income statement, 2 are used in the cash flow statement, and others are used in the footnotes. When rendering the cash flow statement, if I look at all the available contexts for this fact, it would appear there are 4 valid ones, but only 2 of them are actually used in the cash flow statement.

[manusimidt]

Ah yes, I see.

For the income statement, the four concepts would be directly mappable:

However, it's true that the cash flow statement is less detailed regarding the time domain and only two are used.

Unfortunately, I currently don't see a nice/clean way of solving this issue because the presentation linkbase only describes the structure of the financial statements and not if you should state them for 3, 6, 9 or 12 months.

I would try the following approaches:

1. (easiest) always assume that the cash flow statement only uses facts with timespans over 9 months
2. (little bit harder) collect all facts that could be used for the cash flow statement. Separate them, so that all facts that correspond to the same timeframe are in different arrays. Define a threshold, how many facts you at minimum need to construct the cash flow statement. Discard all the facts for the timeframe for which you have to few data.

[manusimidt]

I also added #110 to be able to test for equality between facts and discard duplicate facts easier. Will implement this if I find some time.

[stkerr]

@manusimidt thanks for the reply. That 2nd approach is what I've been doing so far with "reasonable" success. It felt hacky though so wanted to ask if I was missing something, but sounds like that may be what has to happen. I approach it by looking at all contexts that have sets of facts that could meet a given presentation dictionary and then excluding those contexts that don't meet a threshold of facts shared with the context that has the most facts. Currently 50% for the cutoff value seems to work for most filings. It has to be lower than 100% unfortunately, since some filings will omit a fact for a given context and expect the filing to be rendered with a - in its place.

[rayniervanegmond]

Hi Sam A short intro - used to help companies build XBRL processors and reporting solutions as an external consultant: did so for some 20 years almost. In the last couple of years I drifted away from pure XBRL work to do work in AI/Deep Learning in Earth Observation, Satellite image processing and Climate change work with possible connections to ESG reporting. The market for XBRL was drying up for one-man shops in this space So decided to look into something new. If you want I can try to help you out a bit here… I’m a bit rusty on the material but if you want to connect I’m at raynierx at gmail.com Take care - René Raynier van Egmond XBRL Consulting Partners LLC — I’m on LinkedIn too—

…

On Fri, Mar 17, 2023 at 7:03 AM Sam Kerr ***@***.***> wrote: Thanks for the reply! That helps. I updated my code & it can handle dimensions on the same fact reasonably well - it can differentiate Revenue (Total) from Revenue (Hardware/Software/etc). What I'm stuck at with that approach though is that the same fact name may be present multiple times & used in different parts of the statement, such as in the cash flow statement and then later in a footnote. I'm not sure how to differentiate which version to use, which means that I will be returning the wrong one & looking at a context that is for a footnote-only and not a cash flow statement. For a more concrete, consider this 10-Q <https://www.sec.gov/ix?doc=/Archives/edgar/data/2488/000000248822000170/amd-20220924.htm> from AMD. There are 26 results for the fact net income (loss) attributable to parent. 4 of them are used in the income statement, 2 are used in the cash flow statement, and others are used in the footnotes. When rendering the cash flow statement, if I look at all the available contexts for this fact, it would appear there are 4 valid ones, but only 2 of them are actually used in the cash flow statement. — Reply to this email directly, view it on GitHub <#109 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKYVRLR72AW7SGNGPJVDUKLW4RVMPANCNFSM6AAAAAAVYJNRYA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[stkerr]

Hi @rayniervanegmond ! This comment above shows a specific example of what I am trying to solve for, in case you have any ideas around it.

[rayniervanegmond]

Hi - just a quick reminder from what would have been mentioned in the publications you have mentioned above about the structure of "XBRL filings". Each filing consists of multiple files; * Firstly there is the taxonomy which is a set of files that consists of a *XML schema file* and a number of *linkbase files* (XLink files) (one for presentation, one for calculation and one for the multidimensional aspects of the reported data. * Then there is the* instance document* (a regular XML document that references its associated taxonomy XML schema file) which reports the facts for the filing. There is really no way around using ALL THESE FILES to process an XBRL or iXBRL filing. This is why "XBRL processors" like the Py-xbrl ithat Manual has created are so important; they take care of all the parsing and stitching together of the various physical files that make up the "XBRL document". It simply requires a good deal of understanding of the structure and role of all the pieces of the XBRL spec. The XBRL filings that are reported to say the SEC or some other regulator are not a single XML file as a plain vanilla XML-instance/XML Schema scario would be --- its much more complex (though not complicated ;-) than that. Wrt to the roles of these various file: the XBRL Taxonomy *schema file* is a required document because it functions as the data dictionary for all the accounting elements that are used in a filing (e.g. 10-K). If one simply processes just the presentation linkbase file none of the data dictionary "concept definitions" are there and the process becomes nonsensical in XBRL terms. You must process the s*chema file + linkbase files* and if the reported facts have to presented also then *the instance document must also be provided!* If you want some more guidance sent me an email at raynierx at gmail.com take care René

…

On Sun, Apr 30, 2023 at 10:42 AM pnatusch ***@***.***> wrote: While I'm an experienced programmer, I'm new to XBRL and I've been struggling trying to get a handle on it. I've read XBRL for dummies and The XBRL book by Ghislain Fourny. Based on Chapter 5 (Presentation) in "The XBRL book", I got the understanding that from the presentation linkbase that I could get the correct ordering and then get the facts from the instance. Does that make sense? As my first "project", I'm just trying to parse the xbrl documents and print out a balance sheet in the correct order. Randomly, I chose to parse the IBM presentation linkbase, however, after parsing, the linkbase object was empty. What am I doing wrong? If this is a bug, I'm happy to help debug, but my assumption is that I'm just doing something stupid, so I didn't want to go down that path linkbase_url: str = ' https://www.sec.gov/Archives/edgar/data/51143/000155837023002376/ibm-20221231_pre.xml ' linkbase: Linkbase = parse_linkbase_url(linkbase_url, LinkbaseType.LABEL, cache) (Pdb) linkbase.type <LinkbaseType.LABEL: 4> (Pdb) linkbase.linkbase_uri ' https://www.sec.gov/Archives/edgar/data/51143/000155837023002376/ibm-20221231_pre.xml ' (Pdb) linkbase.type <LinkbaseType.LABEL: 4> (Pdb) linkbase.linkbase_uri ' https://www.sec.gov/Archives/edgar/data/51143/000155837023002376/ibm-20221231_pre.xml ' (Pdb) linkbase.to_dict() {'standardExtendedLinkElements': []} (Pdb) linkbase.to_simple_dict() {'standardExtendedLinkElements': []} (Pdb) linkbase.extended_links [] (Pdb) First couple lines from presentation linkbase file <link:linkbase xsi:schemaLocation="http://www.xbrl.org/2003/instance http://www.xbrl.org/2003/xbrl-instance-2003-12-31.xsd"> <roleRef xlink:type="simple" xlink:href="ibm-20221231.xsd#StatementConsolidatedStatementOfComprehensiveIncome" roleURI="http://www.ibm.com — Reply to this email directly, view it on GitHub <#109 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKYVRLRV57MCY6ZTU4L2YADXD2QAXANCNFSM6AAAAAAVYJNRYA> . You are receiving this because you were mentioned.Message ID: ***@***.***>