Cannot retrieve data from previous month any more?

[mr-rulez]

When trying to retrieve data for the previous month, e.g. "last 90 days" period of time in Grafana I receive the following error lines in ticktock db log file:

2023-06-28 07:53:32.546 [ERROR] [qexe_task_1] Failed to mmap file /mount_point/tt/data/2023/4/1681171200.1681257600/data.0, errno = 12
2023-06-28 07:53:32.551 [ERROR] [qexe_task_1] Caught exception while performing query.

"Last 30 days" view is working fine and all datapoints show up.
Is there any chance to recover without any backup snapshot available?

[ylin30]

HI, errno = 12 usually indicates out of memory. Would you mind to share how big is your data and available memory in your device? Is this a 32bit or 64bit OS?

"Last 30 days" view is working fine and all datapoints show up.
Is there any chance to recover without any backup snapshot available?

Your data should be ok in data files. Please try to just zoom in to a specific time range to see if query can succeed.

[mr-rulez]

Hi, it is running Raspbian GNU/Linux 10 (buster) 32-bit OS on a Raspberry Pi 3b+, equipped with 1GB and currently 180 MB free RAM.
Please find the disk usage distribution of the data directory:
pi@raspberry:/mount_point/tt/data $ du -sk *
174 1970
11121 2023
4 append.log
1 ticktock.meta

[ylin30]

Your physical data size is 11MB, not too large. It may be a virtual memory limit problem since it is a 32bit OS. We recently fixed a related bug in version 0.11.8. What version of TickTockDB are you running? You can find that in log when starting.

Would you mind to:

1. Try querying, e.g,. just 4/2023 instead of last 3 months, to see if success.
2. run: pmap -x <TT id> > /tmp/pmap.txt, and send the result over.

thanks

[mr-rulez]

Currently, I am using version 0.11.1. How to "upgrade" :) ?

1. Actually, when looking only for month of April, I realized that I was not precise enough: I am missing data from 2023-04-10 01:30:00 (last visible data point) onwards.
   When trying to display month of May 2023 no data is visualized at all!
   But I am able to

visualize from 2023-05-10 02:00:00 until now.

--> And now I found a problem which started with 2023-05-14 resulting in a lot of error lines looking identical except the timestamp:
2023-05-14 03:10:02.608 [ERROR] [tcp_listener_0] fd (273) >= m_min_file (94)

Restart on 2023-05-20 brought up some log lines like "2023-05-20 21:50:02.184 [INFO] [main] tsdb dir /mount_point/tt/data/2023/2/1675209600.1675209600.back ignored during restore_tsdb()
and a single error line which reoccurred yesterday when I reported this issue:
2023-05-20 21:50:15.289 [ERROR] [main] Bad line in /mount_point/tt/data/ticktock.meta: RX_bytes

2. Output is:
   7817: /opt/ticktock/bin/tt -d -c /opt/ticktock/conf/tt.conf

[ylin30]

Currently, I am using version 0.11.1. How to "upgrade" :) ?

Rebuild new src codes or replace the binary in ./bin.

--> And now I found a problem which started with 2023-05-14 resulting in a lot of error lines looking identical except the timestamp: 2023-05-14 03:10:02.608 [ERROR] [tcp_listener_0] fd (273) >= m_min_file (94)

Restart on 2023-05-20 brought up some log lines like "2023-05-20 21:50:02.184 [INFO] [main] tsdb dir /mount_point/tt/data/2023/2/1675209600.1675209600.back ignored during restore_tsdb() and a single error line which reoccurred

It is ok to ignore .back folders. They are original data before compaction.

yesterday when I reported this issue: 2023-05-20 21:50:15.289 [ERROR] [main] Bad line in /mount_point/tt/data/ticktock.meta: RX_bytes

Seem corruption in meta file. This worries me.

If possible, would you mind to share your data in ./data folder with me? Let me repro your problem in my side. It would be much easier.

2. Output is:
   7817: /opt/ticktock/bin/tt -d -c /opt/ticktock/conf/tt.conf

oh, if tt's pid is 7817, I meant to run pmap -x 7817. And send the results to me.

[mr-rulez]

Any private and less public possibility (personal message, etc.) to share the data?

[ylin30]

Please send them to my email [email protected].

[ylin30]

@mr-rulez Did you send the data to my email yet?

Another thing you can try is to replace, e.g., 1529280000.1531872000 dir with 1529280000.1531872000.back. And check if the queries can succeed. Basically, an old TSDB (e.g., 1529280000.1531872000) would be compacted. The original data is backup in 1529280000.1531872000.back. We discovered a problem in compaction which may require huge virtual memory. Even though not a problem in 64bit OS, it may fail large queries in 32bit OS since the max vsize is only 4GB (in practice 2.5GB). Note that the bug we fixed in 0.11.8 only fixes incoming data, but it does not adjust old existing data.

yesterday when I reported this issue: 2023-05-20 21:50:15.289 [ERROR] [main] Bad line in /mount_point/tt/data/ticktock.meta: RX_bytes

I am more worried about this problem.