WIP: Store compiled modules #44
Conversation
Modules are compiled and stored in a content addressable store. Two runs of the same module are significantly faster since we only need to digest the wasm module being loaded and do not need to compile it again. This is still a work in progress. The store is not really concurrency safe yet (it mostly works, but multiple threads/processes could hit issues). Signed-off-by: Brian Goff <[email protected]>
| let mut file = File::open(path).context("could not open metadata file")?; | ||
| let mut buf = String::new(); | ||
| file.read_to_string(&mut buf) | ||
| .context("could not read metadata file")?; | ||
|
|
||
| Ok(serde_json::from_str(&buf).context("could not parse metadata file")?) |
There was a problem hiding this comment.
This could be written like
let data = fs::read_to_string(path).context("could not read metadata file");
Ok(serde_json::from_str(&data).context("could not parse metadata file")?)fs::read_to_string does the exact same thing your code does
|
|
||
| impl ContentWriter { | ||
| pub fn write(&mut self, data: &[u8]) -> Result<usize> { | ||
| self.f.try_clone()?.write(data).context("write") |
There was a problem hiding this comment.
Curious why do you try_clone here? Can't we directly write self.f.write(data).context("write")?
There was a problem hiding this comment.
Lots of issues with borrowing if I recall correctly. Could also just be a remnant of me trying to get a different implementation working with file locking.
|
@cpuguy83 does this make any allowance for precise understanding of the module to check whether it's been cached? That is, it seems like it checks a sha.... which I'd be assuming is a digest. Am I right? Just want to make sure it's easy to move from a quick, possibly unsafe check to a more formal one down the line.... looks GREAT though. |
|
@squillace This is based on the digest of the module from the image. |
|
I'm wondering whether that's really just a load problem: if you pull an arm image on an amd node, you'll get the same "I just can't today" error. Though if there's a coherent way to determine and handle it in the future..... |
|
With the wasm oci work, this becomes possible and I've been working a bit on testing it out. I'll open a separate that does something similiar. |
Modules are compiled and stored in a content addressable store. Two runs of the same module are significantly faster since we only need to digest the wasm module being loaded and do not need to compile it again.
This is still a work in progress.
The store is not really concurrency safe yet (it mostly works, but multiple threads/processes could hit issues).