Build IndexedDB schema

[hitchhooker]

Summary

We need to develop an enhanced database schema for our application that efficiently manages .polkavm files, particularly focusing on handling memory and disassembly data, along with file metadata. This schema should support storing large files in a chunked manner and provide mechanisms for tracking completeness and integrity of the data.

Goals

• Create separate object stores for memory data, disassembly data, and file metadata.
• Implement logic for chunk management to handle large files.
• Develop mechanisms for tracking and verifying the completeness of chunked data.

Detailed Description

Our current database schema requires modifications to handle large .polkavm files more effectively. The proposed schema will include:

1. Object Stores:

   • file_metadata: To store metadata about each file, including information on chunk count and processing status.
   • memory_data: For storing the memory portion of the .polkavm files.
   • disassembly_data: To store disassembly data in chunks.

2. Metadata Structure:

   • The FileMetadata struct should include file_id, total_chunks, processed_chunks, and is_complete fields.
   • This will help in tracking the number of chunks processed and determining when a file's processing is complete.

3. Chunk Processing and Tracking:

   • Implement functionality to add and retrieve chunks of disassembly data.
   • Update file metadata after each chunk addition to keep track of the processing progress.

4. Data Integrity Checks:

   • Consider implementing checksums for each chunk and the entire file.
   • Develop a reconciliation process to run after all chunks are processed to ensure data integrity.

Tasks

• Modify the database schema to include new object stores and structures.
• Implement functions for adding and retrieving data to/from these stores.
• Create logic for updating and checking file metadata based on chunk processing.
• Write unit tests for new database functionalities.

Potential Challenges

• Ensuring data integrity with concurrent chunk processing.
• Efficiently handling large files without impacting application performance.

Resources

• Rexie Documentation
• IndexedDB API