Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: acme_fcase_trace negative counter values #12

Open
nawtrey opened this issue Dec 16, 2021 · 1 comment
Open

BUG: acme_fcase_trace negative counter values #12

nawtrey opened this issue Dec 16, 2021 · 1 comment

Comments

@nawtrey
Copy link
Contributor

nawtrey commented Dec 16, 2021

It looks like the acme_fcase_trace log has some negative data for the POSIX_F_META_TIME counter. I noticed this while attempting to write a regression test for issue #590.

Here is some output illustrating the problem:

module:  POSIX
Negative values:  [-3.06159650e+171 -1.72714531e-307 -8.63423493e-307 -4.79982338e-306]
by-average data: 
Read      1.624701e-01
Write     4.472180e-02
Meta    -3.737300e+167
----------------------
module:  MPIIO
Negative values:  []
by-average data: 
Read     0.157147
Write    0.436347
Meta     0.003968
----------------------
module:  STDIO
Negative values:  []
by-average data: 
Read     0.067240
Write    0.000083
Meta     0.042183
----------------------
Final I/O cost dataframe:
           Read     Write           Meta
POSIX  0.162470  0.044722 -3.737300e+167
MPIIO  0.157147  0.436347   3.967560e-03
STDIO  0.067240  0.000083   4.218273e-02

Basically the POSIX_F_META_TIME counter has a handful of negative values, one of which is very large, and this is resulting in a large, negative sum for the Meta time. Maybe @shanedsnyder can take a quick look at the counter data to confirm, but for now I'll probably look at using a different log.

@carns
Copy link
Contributor

carns commented Dec 17, 2021

Wow, that field is really way off:

POSIX   -1      2990992946131980000     POSIX_F_META_TIME       -3061596495762055327894679314511836560340477487190259516241843015484138233235383091501060635977155859197306952421967514327453521398186444153083051710408022781116533717336064.000000    /gpfs/mira-fs1/projects/radix-io/snyder/acme/test_F_case_cetus_dxt/run/drv_flds_in      /gpfs/mira-fs1  gpfs

The rest of the record for that drv_flds_in file doesn't look particularly remarkable, though. It was opened 8K times and read 16K times.

I'm not coming up with many ideas for how to work out what went wrong here from looking at the log.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants