-
Notifications
You must be signed in to change notification settings - Fork 654
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
stream load. Is it possible? #565
Comments
I am not sure, but can we pass io.BytesIO as std::ifstream? Line 152 in 3f34296
void loadIndex(const std::ifstream &input, SpaceInterface<dist_t> *s) {
std::streampos position;
readBinaryPOD(input, maxelements_);
readBinaryPOD(input, size_per_element_);
readBinaryPOD(input, cur_element_count);
data_size_ = s->get_data_size();
fstdistfunc_ = s->get_dist_func();
dist_func_param_ = s->get_dist_func_param();
size_per_element_ = data_size_ + sizeof(labeltype);
data_ = (char *) malloc(maxelements_ * size_per_element_);
if (data_ == nullptr)
throw std::runtime_error("Not enough memory: loadIndex failed to allocate data");
input.read(data_, maxelements_ * size_per_element_);
input.close();
} |
split function at first phase void loadStream(const std::ifstream &input, SpaceInterface<dist_t> *s) {
readBinaryPOD(input, maxelements_);
readBinaryPOD(input, size_per_element_);
readBinaryPOD(input, cur_element_count);
data_size_ = s->get_data_size();
fstdistfunc_ = s->get_dist_func();
dist_func_param_ = s->get_dist_func_param();
size_per_element_ = data_size_ + sizeof(labeltype);
data_ = (char *) malloc(maxelements_ * size_per_element_);
if (data_ == nullptr)
throw std::runtime_error("Not enough memory: loadIndex failed to allocate data");
input.read(data_, maxelements_ * size_per_element_);
}
void loadIndex(const std::string &location, SpaceInterface<dist_t> *s) {
std::ifstream input(location, std::ios::binary);
std::streampos position;
loadStream(input, s);
input.close();
} |
the same things with it Line 716 in 3f34296
|
Unfortunately, we can't just do this because functions are used. .seekg() and .tellg() (we can simplify loading code and remove it) and maybe std::ifstream is not compatible with io.ByteIO and we need std::istringstream What do you think about? |
Take a look at #556 |
I want to load network resourse to index but it failed
got error
Is it normal idea?
The text was updated successfully, but these errors were encountered: