[Bug] Can't upload file to VirusTotal

[sccoouut]

Hi, i'm using VirusTotal API in order to check is file malware, so I need to upload file. I tried to do it in that way, but it have not succeeded:

httplib::Client cli("https://www.virustotal.com");

httplib::Headers headers{
	{ "Accept", "text/plain" },
	{ "Content-Type", "application/x-www-form-urlencoded" }
};
httplib::MultipartFormDataItems items{
	{ "apikey", api_key, "", "" },
	{ "file", std::string(buffer.begin(), buffer.end()), file_name, ""}
};

auto res = cli.Post("/vtapi/v2/file/scan", headers, items);

The error that I get is: "Write (5)"
I tried to do it also in another way and it have not worked too:

httplib::Client cli("https://www.virustotal.com");

httplib::Headers headers{
	{ "Accept", "text/plain" },
	{ "Content-Type", "application/x-www-form-urlencoded" }
};
httplib::Params params{
	{ "apikey", std::string(api_key) },
	{ "file", "data:name=" + std::string(file_path) + ";base64," + encoded }
};

auto res = cli.Post("/vtapi/v2/file/scan", headers, params);

Although, I tried to use curl lib and it works great, there is a working code:

auto params = "apikey=" + std::string(api_key) + "&file=data:name=" + std::string(file_path) + ";base64," + encoded;

CURL* hnd = curl_easy_init();
curl_easy_setopt(hnd, CURLOPT_CUSTOMREQUEST, "POST");
curl_easy_setopt(hnd, CURLOPT_URL, "https://www.virustotal.com/vtapi/v2/file/scan");
struct curl_slist* headers = NULL;
headers = curl_slist_append(headers, "Accept: text/plain");
headers = curl_slist_append(headers, "Content-Type: application/x-www-form-urlencoded");
curl_easy_setopt(hnd, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(hnd, CURLOPT_POSTFIELDS, params.c_str());
std::string readBuffer;
curl_easy_setopt(hnd, CURLOPT_WRITEFUNCTION, WriteCallback);
curl_easy_setopt(hnd, CURLOPT_WRITEDATA, &readBuffer);
CURLcode ret = curl_easy_perform(hnd);
curl_easy_reset(hnd);

Am I doing something wrong?

[PixlRainbow]

did you compile httplib with openssl enabled?

[sccoouut]

yep. it even worked with small file, but with big files it does not work

[yhirose]

it even worked with small file, but with big files it does not work

If it works with a small file, I don't think it's a problem of cpp-httplib, but you simply hit a size limit on the service that you are using. Could you verify that?

[sccoouut]

it even worked with small file, but with big files it does not work

If it works with a small file, I don't think it's a problem of cpp-httplib, but you simply hit a size limit on the service that you are using. Could you verify that?

The second file has worked with the cURL library.

[yhirose]

@sccoouut, could you give me more detailed information about the 'second file'?

[sccoouut]

@sccoouut, could you give me more detailed information about the 'second file'?

The first file (that worked) is .txt file with one word. The second file is cpuminer last release. I have tested my code on Windows 10 with MSVC and VS22 (the httplib and needed depends I have install with vcpkg).

[yhirose]

@sccoouut, I am actually concerned with the size of the second file, not the type of it. Could you figure out the minimum size that can cause the problem, so that I could easily reproduce it on my machine.

Could you also give more detail about the difference between the actual HTTP request data from cpp-httplib and the one from cURL? (You can get the info with Client::set_logger. As for CURL, I think you can use the command-line version of curl with option -v.)

cpp-httplib/httplib.h

Line 1030 in 65a8f4c

void set_logger(Logger logger);

Thanks a lot!

[yhirose]

any response?

[sccoouut]

Hi, I have been a little bit busy on the last week, I'll do it in few next days.

[yhirose]

Close it until further information is provided.

[sccoouut]

When I try to install logger on my client, it's not being called. Am I doing something wrong?

void logger( const httplib::Request &request, const httplib::Response &response ) {
	std::cout << request.body << std::endl;
}

std::ifstream file( file_path, std::ios::binary );
std::vector<uint8_t> file_data( std::istreambuf_iterator<char>( file ), {} );
file.close();

httplib::Client cli( "https://www.virustotal.com" );
cli.set_logger( logger );
httplib::Headers headers{
	{ "Accept", "text/plain" },
	{ "Content-Type", "application/x-www-form-urlencoded" }
};
httplib::MultipartFormDataItems items{
	{ "apikey", api_key, "", "" },
	{ "file", std::string( file_data.begin(), file_data.end() ), file_name, "" } 
};

auto res = cli.Post( "/vtapi/v2/file/scan", headers, items );
std::cout << res->body << std::endl;

[sccoouut]

@yhirose

[yhirose]

Working example:

#include <httplib.h>
#include <iostream>

int main(void) {
  httplib::Client cli("https://httpbin.org");

  cli.set_logger([](const httplib::Request &req, const httplib::Response &res) {
    std::cout << req.body << std::endl;
    std::cout << res.body << std::endl;
  });

  auto res = cli.Post("/post", "hello world!", "text/plain");
}

Result:

hello world!
{
  "args": {},
  "data": "hello world!",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Content-Length": "12",
    "Content-Type": "text/plain",
    "Host": "httpbin.org",
    "User-Agent": "cpp-httplib/0.10.2",
    "X-Amzn-Trace-Id": "Root=1-620d740b-194db0ce619499692cbca2a1"
  },
  "json": null,
  "origin": "208.74.141.25",
  "url": "https://httpbin.org/post"
}

[PixlRainbow]

There is a problem. If there is a connection failure, logger will not be called because ClientImpl::process_request will return immediately (before it reaches the logger call):

cpp-httplib/httplib.h

Lines 6354 to 6435 in bb00a23

	inline bool ClientImpl::process_request(Stream &strm, Request &req,
	Response &res, bool close_connection,
	Error &error) {
	// Send request
	if (!write_request(strm, req, close_connection, error)) { return false; }
	// Receive response and headers
	if (!read_response_line(strm, req, res) ||
	!detail::read_headers(strm, res.headers)) {
	error = Error::Read;
	return false;
	}
	// Body
	if ((res.status != 204) && req.method != "HEAD" && req.method != "CONNECT") {
	auto redirect = 300 < res.status && res.status < 400 && follow_location_;
	if (req.response_handler && !redirect) {
	if (!req.response_handler(res)) {
	error = Error::Canceled;
	return false;
	}
	}
	auto out =
	req.content_receiver
	? static_cast<ContentReceiverWithProgress>(
	[&](const char *buf, size_t n, uint64_t off, uint64_t len) {
	if (redirect) { return true; }
	auto ret = req.content_receiver(buf, n, off, len);
	if (!ret) { error = Error::Canceled; }
	return ret;
	})
	: static_cast<ContentReceiverWithProgress>(
	[&](const char *buf, size_t n, uint64_t /*off*/,
	uint64_t /*len*/) {
	if (res.body.size() + n > res.body.max_size()) {
	return false;
	}
	res.body.append(buf, n);
	return true;
	});
	auto progress = [&](uint64_t current, uint64_t total) {
	if (!req.progress || redirect) { return true; }
	auto ret = req.progress(current, total);
	if (!ret) { error = Error::Canceled; }
	return ret;
	};
	int dummy_status;
	if (!detail::read_content(strm, res, (std::numeric_limits<size_t>::max)(),
	dummy_status, std::move(progress), std::move(out),
	decompress_)) {
	if (error != Error::Canceled) { error = Error::Read; }
	return false;
	}
	}
	if (res.get_header_value("Connection") == "close" ||
	(res.version == "HTTP/1.0" && res.reason != "Connection established")) {
	// TODO this requires a not-entirely-obvious chain of calls to be correct
	// for this to be safe. Maybe a code refactor (such as moving this out to
	// the send function and getting rid of the recursiveness of the mutex)
	// could make this more obvious.
	// This is safe to call because process_request is only called by
	// handle_request which is only called by send, which locks the request
	// mutex during the process. It would be a bug to call it from a different
	// thread since it's a thread-safety issue to do these things to the socket
	// if another thread is using the socket.
	std::lock_guard<std::mutex> guard(socket_mutex_);
	shutdown_ssl(socket_, true);
	shutdown_socket(socket_);
	close_socket(socket_);
	}
	// Log
	if (logger_) { logger_(req, res); }
	return true;
	}

[yhirose]

@PixlRainbow, thanks for the clarification. Yes, when a connection error happens, the logger won't be called, since the logger is HTTP level logger.

@sccoouut, hope @PixlRainbow's comment helps you understand the situation. TCP/IP level error logger isn't available, yet. (#870)

[PixlRainbow]

it also makes it impossible to identify what the difference is between httplib's request data and curl's request data that makes the virustotal request fail
I suppose as an alternative you could attach a debugger and mark a breakpoint where process_request is called and then inspect the request object