Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Client Transport #215

Merged
merged 126 commits into from
Jan 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
126 commits
Select commit Hold shift + click to select a range
72da252
First cut at standalone TCP transport
Philip-NLnetLabs Dec 19, 2022
a9895ee
Added TODOs and a comment about predictable IDs
Philip-NLnetLabs Jan 2, 2023
2c02b38
Improved text about random IDs
Philip-NLnetLabs Jan 2, 2023
99fe4fc
Separate Query object to capture the lifetime of the query.
Philip-NLnetLabs Feb 20, 2023
eeace82
Support for idle timeouts and edns-tcp-keepalive
Philip-NLnetLabs Feb 27, 2023
d7baa64
Handle tcp read and write errors
Philip-NLnetLabs Mar 7, 2023
544933a
A bit of cleanup
Philip-NLnetLabs Mar 7, 2023
cf76e57
Timeouts and a bit of cleanup
Philip-NLnetLabs Mar 13, 2023
297d5eb
Limit the number of concurrent queries to 32K and some small changes
Philip-NLnetLabs Mar 15, 2023
1500676
Rewrote tokio::select! loop
Philip-NLnetLabs Mar 17, 2023
24ba9a6
Rename transport to net.
Philip-NLnetLabs Mar 30, 2023
1b4244c
Renamed tcp.rs to tcp_mutex.ts and fixed clippy warnings.
Philip-NLnetLabs Apr 17, 2023
6d004b6
A channel based implementation of a TCP transport.
Philip-NLnetLabs Apr 17, 2023
52aa7b0
Fix some clippy warnings
Philip-NLnetLabs Apr 17, 2023
2ea565e
Fix fmt errors.
Philip-NLnetLabs Apr 17, 2023
83dc4aa
keep making changes until both fmt and clippy are happy.
Philip-NLnetLabs Apr 17, 2023
4e8d0a3
Make tcp_channel.rs a bit more general. Now octet_stream.rs.
Philip-NLnetLabs Apr 18, 2023
63b2030
Fix fmt problem.
Philip-NLnetLabs Apr 18, 2023
8d7fe2b
Added example.
Philip-NLnetLabs Apr 19, 2023
fef193b
Run now takes a single 'io' object. Separate TCP and TLS examples.
Philip-NLnetLabs Apr 19, 2023
7f455e4
Require 'net' feature for tcp-client and tls-client examples.
Philip-NLnetLabs Apr 19, 2023
c2b1cdd
Separate function add_tcp_keepalive to add the edns-tcp-keepalive option
Philip-NLnetLabs Apr 25, 2023
2809889
Adjust to new opt interface
Philip-NLnetLabs May 8, 2023
331d674
query_no_check (don't check if answer match query), is_answer_ignore_…
Philip-NLnetLabs Jun 30, 2023
99fcb0d
Trait for connection factories
Philip-NLnetLabs Jun 30, 2023
7414791
TCP connection factory
Philip-NLnetLabs Jun 30, 2023
7774f0c
TLS connection factury
Philip-NLnetLabs Jun 30, 2023
29cf98f
A DNS over multiple octet streams transport
Philip-NLnetLabs Jun 30, 2023
e49b2c3
Add factory, multi_stream, tcp_factory, tls_factory.
Philip-NLnetLabs Jun 30, 2023
c268437
Add tokio-rustls and add rt-multi-thread to tokio
Philip-NLnetLabs Jun 30, 2023
7ca9893
Disable imm_retry_count for the moment
Philip-NLnetLabs Jun 30, 2023
3f98ce7
replace unwrap with expect and layout
Philip-NLnetLabs Jul 3, 2023
eec1f85
Ignore existing TcpKeepalive
Philip-NLnetLabs Jul 20, 2023
fc80f80
UDP and UDP+TCP client transports
Philip-NLnetLabs Aug 7, 2023
bc60041
Traits for queries and an error type.
Philip-NLnetLabs Aug 9, 2023
e65218a
Add Send
Philip-NLnetLabs Aug 10, 2023
4979a62
Fmt
Philip-NLnetLabs Aug 10, 2023
fd92f07
QueryMessage2 and Debug
Philip-NLnetLabs Sep 21, 2023
0eff109
A transport that multiplexes requests over multiple redundant transports
Philip-NLnetLabs Sep 21, 2023
6e2a907
Redo examples.
Philip-NLnetLabs Sep 21, 2023
3302a3c
Cargo fmt.
Philip-NLnetLabs Sep 21, 2023
35170b7
Set ID in the header to new random value each time before sending a r…
Philip-NLnetLabs Oct 3, 2023
80f8479
Introduction of QueryMessage3, which takes a Message instead of a
Philip-NLnetLabs Oct 4, 2023
3e0f0d8
Updated client-transports example.
Philip-NLnetLabs Oct 4, 2023
3e564ba
Get rid of QueryMessage2
Philip-NLnetLabs Oct 4, 2023
a68d623
Change the query_msg in QueryMessage from MessageBuilder to Message.
Philip-NLnetLabs Oct 4, 2023
4462d7e
tcp_channel has been replaced by octet_stream
Philip-NLnetLabs Oct 4, 2023
25fe7a6
Get rid of most unwraps.
Philip-NLnetLabs Oct 9, 2023
086cd28
Clippy
Philip-NLnetLabs Oct 9, 2023
431a7ba
Check result of add.
Philip-NLnetLabs Oct 9, 2023
72c36de
rt-multi-thread is not needed.
Philip-NLnetLabs Oct 9, 2023
1d72b13
Added minor version where major version is 0.
Philip-NLnetLabs Oct 9, 2023
9d6fa92
Get rid of Pin<Box< in factory.
Philip-NLnetLabs Oct 9, 2023
1ab08b6
Feature net depends on std.
Philip-NLnetLabs Oct 9, 2023
9d7be4c
Reorder and section markers
Philip-NLnetLabs Oct 10, 2023
9ff795b
New GetResult using a nested future to be cancel-safe
Philip-NLnetLabs Oct 10, 2023
cc72dc0
Remove tcp_mutex. This was an implementation using shared state (inst…
Philip-NLnetLabs Oct 10, 2023
fc4c11d
Add configuration options.
Philip-NLnetLabs Oct 16, 2023
d3bc706
Continue receiving if there is an error. Allow an empty question sect…
Philip-NLnetLabs Oct 19, 2023
9dad322
Fix a bug where a message with the wrong ID is returned.
Philip-NLnetLabs Oct 19, 2023
9f0c404
Allow the question section to be empty if the reply is an error.
Philip-NLnetLabs Oct 19, 2023
15c0c5c
User configuration.
Philip-NLnetLabs Oct 19, 2023
63d7f46
Update client-transports example.
Philip-NLnetLabs Oct 19, 2023
6947f56
Updated version of tokio and proc-macro2.
Philip-NLnetLabs Oct 19, 2023
ff37fc7
Switch to BaseMessageBuilder.
Philip-NLnetLabs Nov 6, 2023
5f72cdf
derive Debug
Philip-NLnetLabs Nov 6, 2023
f46e419
Make sure that dropping all references to connection objects terminat…
Philip-NLnetLabs Nov 16, 2023
7fafb1d
Less debug output.
Philip-NLnetLabs Dec 1, 2023
b0e3024
Make sure run tasks terminate. Timeouts to avoid waiting forever.
Philip-NLnetLabs Dec 1, 2023
07b9be7
Handle UDP payload size.
Philip-NLnetLabs Dec 4, 2023
92b7546
Simpify message building, add udp_payload_size to UDP config.
Philip-NLnetLabs Dec 4, 2023
c121bf0
Use into_message.
Philip-NLnetLabs Dec 4, 2023
c770a38
Better error handling for TCP connection.
Philip-NLnetLabs Dec 4, 2023
b64418a
We need 0.21.9 for rustls to pass minimal version tests
Philip-NLnetLabs Dec 5, 2023
33d73f8
Rename ConnFactory, TcpConnFactory, TlsConnFactory to ConnectionStream,
Philip-NLnetLabs Dec 5, 2023
2151e12
Rename ConnectionStream to AsyncConnect
Philip-NLnetLabs Dec 7, 2023
552ab5c
Move parameter IO to associated type Connection.
Philip-NLnetLabs Dec 7, 2023
f186a79
Remove type constraints for Connection.
Philip-NLnetLabs Dec 7, 2023
53803b5
Rename BaseMessageBuilder, BMB to ComposeRequest, RequestMessage.
Philip-NLnetLabs Dec 7, 2023
a04da47
Rename QueryMessage4, GetResult to Request, GetResponse.
Philip-NLnetLabs Dec 8, 2023
1e9487d
Add remark about the need for associated types in the future.
Philip-NLnetLabs Dec 8, 2023
8147c8b
Use crate::net::client transports.
Philip-NLnetLabs Nov 20, 2023
a4f04ea
Reduce the number of tasks we need.
Philip-NLnetLabs Nov 17, 2023
effbd2a
Factory is renamed to connection stream
Philip-NLnetLabs Dec 5, 2023
6a5350c
Adapt to renaming in net/client
Philip-NLnetLabs Dec 8, 2023
34ecd44
Get rid of ign_tc.
Philip-NLnetLabs Dec 8, 2023
b22e2f4
Merge branch 'main' into transport
partim Dec 20, 2023
565ea77
Fix features and imports.
partim Dec 20, 2023
d6f4491
Fix format.
partim Dec 20, 2023
5f22519
Move all transport protocol stuff into a single module.
partim Dec 21, 2023
01056b6
Move things around. Have RequestMessage keep an OptRecord.
partim Dec 21, 2023
a895038
Rename Request to SendRequest.
partim Dec 21, 2023
36ce917
Introduce unstable features.
partim Dec 28, 2023
167c0e8
Initial tests for net::client
Philip-NLnetLabs Dec 22, 2023
dd4e236
Datagram transports. This abstracts from UDP.
Philip-NLnetLabs Dec 27, 2023
5f01c4f
Test for dgram.
Philip-NLnetLabs Dec 28, 2023
4e6f4cd
Introduce dgram_stream to replace udp_tcp
Philip-NLnetLabs Dec 28, 2023
f2bbdf7
Test for dgram_stream.
Philip-NLnetLabs Dec 28, 2023
b42c6b6
Switch to dgram_stream
Philip-NLnetLabs Dec 28, 2023
240bdfc
Remove udp and udp_tcp.
Philip-NLnetLabs Dec 28, 2023
5217e16
Fmt
Philip-NLnetLabs Dec 28, 2023
eb13944
Rename octet_stream to just stream.
partim Dec 28, 2023
059a779
Reflow.
partim Dec 28, 2023
bb4a0ed
Poll-based AsyncDgramRecv and AsyncDgramSend and helper traits for
Philip-NLnetLabs Dec 28, 2023
bacdca9
More documentation for net::client.
Philip-NLnetLabs Dec 29, 2023
598a886
Refactor stream transports to provide a use-once run future.
partim Jan 2, 2024
b0225f0
Clippy-suggested fixes.
partim Jan 2, 2024
7703680
Add unstable-client-transport to resolv.
partim Jan 2, 2024
0e6ec78
Fix doctests.
partim Jan 2, 2024
acbb06c
Simplify stream transport.
partim Jan 3, 2024
db61838
We need Slab 0.4.9 not 0.4.0.
partim Jan 3, 2024
684564f
Bring back our own slab.
partim Jan 3, 2024
9ba57d3
Format.
partim Jan 3, 2024
2a3c615
Simplify all the other basic transports as well.
partim Jan 4, 2024
e4f7049
Don’t run the client doc examples.
partim Jan 4, 2024
b9b733a
Further improvements to the datagram client.
partim Jan 5, 2024
31a1117
Mark the big example as no_run.
partim Jan 5, 2024
8d2da50
Change the datagram transmit loop into a for loop.
partim Jan 8, 2024
a325784
Impl Display and Error for request::Error.
partim Jan 9, 2024
548d239
Revert back to the previous design.
partim Jan 9, 2024
5a9f862
Change creation of redundant transport to new model.
partim Jan 9, 2024
1ed6117
write_all is not cancel safe, use write instead. Change deckard to ac…
Philip-NLnetLabs Jan 11, 2024
bfb5399
Get rid of unwrap.
Philip-NLnetLabs Jan 11, 2024
54021f1
Improve documentation
Philip-NLnetLabs Jan 12, 2024
a08366e
Remove TryQueryError and remove pub from handle_request_impl.
Philip-NLnetLabs Jan 12, 2024
0fd561d
Merge branch 'main' into transport
partim Jan 15, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 20 additions & 5 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,9 @@ name = "domain"
path = "src/lib.rs"

[dependencies]
octseq = { version = "0.3.2", default-features = false }
time = { version = "0.3.1", default-features = false }
octseq = { version = "0.3.2", default-features = false }
pin-project-lite = "0.2"
time = { version = "0.3.1", default-features = false }

rand = { version = "0.8", optional = true }
bytes = { version = "1.0", optional = true, default-features = false }
Expand All @@ -30,7 +31,11 @@ ring = { version = "0.17", optional = true }
serde = { version = "1.0.130", optional = true, features = ["derive"] }
siphasher = { version = "1", optional = true }
smallvec = { version = "1", optional = true }
tokio = { version = "1.0", optional = true, features = ["io-util", "macros", "net", "time"] }
tokio = { version = "1.33", optional = true, features = ["io-util", "macros", "net", "time", "sync", "rt-multi-thread" ] }
tokio-rustls = { version = "0.24", optional = true, features = [] }

# XXX Force proc-macro2 to at least 1.0.69 for minimal-version build
proc-macro2 = "1.0.69"

[target.'cfg(macos)'.dependencies]
# specifying this overrides minimum-version mio's 0.2.69 libc dependency, which allows the build to work
Expand All @@ -41,24 +46,31 @@ default = ["std", "rand"]
bytes = ["dep:bytes", "octseq/bytes"]
heapless = ["dep:heapless", "octseq/heapless"]
interop = ["bytes", "ring"]
resolv = ["bytes", "futures-util", "smallvec", "std", "tokio", "libc", "rand"]
resolv = ["net", "smallvec", "std", "rand", "unstable-client-transport"]
resolv-sync = ["resolv", "tokio/rt"]
serde = ["dep:serde", "octseq/serde"]
sign = ["std"]
smallvec = ["dep:smallvec", "octseq/smallvec"]
std = ["bytes?/std", "octseq/std", "time/std"]
net = ["bytes", "futures-util", "std", "tokio", "tokio-rustls"]
tsig = ["bytes", "ring", "smallvec"]
validate = ["std", "ring"]
zonefile = ["bytes", "std"]

# Unstable features
unstable-client-transport = []

# This feature should include all features that the CI should include for a
# test run. Which is everything except interop.
ci-test = ["resolv", "resolv-sync", "sign", "std", "serde", "tsig", "validate", "zonefile"]
ci-test = ["net", "resolv", "resolv-sync", "sign", "std", "serde", "tsig", "validate", "zonefile"]

[dev-dependencies]
rustls = { version = "0.21.9" }
serde_test = "1.0.130"
serde_yaml = "0.9"
tokio = { version = "1", features = ["rt-multi-thread", "io-util", "net"] }
tokio-test = "0.4"
webpki-roots = { version = "0.25" }

[package.metadata.docs.rs]
all-features = true
Expand All @@ -84,3 +96,6 @@ required-features = ["resolv-sync"]
name = "client"
required-features = ["std", "rand"]

[[example]]
name = "client-transports"
required-features = ["net"]
233 changes: 233 additions & 0 deletions examples/client-transports.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,233 @@
/// Using the `domain::net::client` module for sending a query.
use domain::base::Dname;
use domain::base::MessageBuilder;
use domain::base::Rtype::Aaaa;
use domain::net::client::dgram;
use domain::net::client::dgram_stream;
use domain::net::client::multi_stream;
use domain::net::client::protocol::{TcpConnect, TlsConnect, UdpConnect};
use domain::net::client::redundant;
use domain::net::client::request::{RequestMessage, SendRequest};
use domain::net::client::stream;
use std::net::{IpAddr, SocketAddr};
use std::str::FromStr;
use std::time::Duration;
use tokio::net::TcpStream;
use tokio::time::timeout;
use tokio_rustls::rustls::{ClientConfig, OwnedTrustAnchor, RootCertStore};

#[tokio::main]
async fn main() {
// Create DNS request message.
//
// Transports currently take a `RequestMessage` as their input to be able
// to add options along the way.
//
// In the future, it will also be possible to pass in a message or message
// builder directly as input but for now it needs to be converted into a
// `RequestMessage` manually.
let mut msg = MessageBuilder::new_vec();
msg.header_mut().set_rd(true);
let mut msg = msg.question();
msg.push((Dname::vec_from_str("example.com").unwrap(), Aaaa))
.unwrap();
let req = RequestMessage::new(msg);

// Destination for UDP and TCP
let server_addr = SocketAddr::new(IpAddr::from_str("::1").unwrap(), 53);

let mut stream_config = stream::Config::new();
stream_config.set_response_timeout(Duration::from_millis(100));
let multi_stream_config =
multi_stream::Config::from(stream_config.clone());

// Create a new UDP+TCP transport connection. Pass the destination address
// and port as parameter.
let mut dgram_config = dgram::Config::new();
dgram_config.set_max_parallel(1);
dgram_config.set_read_timeout(Duration::from_millis(1000));
dgram_config.set_max_retries(1);
dgram_config.set_udp_payload_size(Some(1400));
let dgram_stream_config = dgram_stream::Config::from_parts(
dgram_config.clone(),
multi_stream_config.clone(),
);
let udp_connect = UdpConnect::new(server_addr);
let tcp_connect = TcpConnect::new(server_addr);
let (udptcp_conn, transport) = dgram_stream::Connection::with_config(
udp_connect,
tcp_connect,
dgram_stream_config,
);

// Start the run function in a separate task. The run function will
// terminate when all references to the connection have been dropped.
// Make sure that the task does not accidentally get a reference to the
// connection.
tokio::spawn(async move {
transport.run().await;
println!("UDP+TCP run exited");
});

// Send a query message.
let mut request = udptcp_conn.send_request(req.clone());

// Get the reply
println!("Wating for UDP+TCP reply");
let reply = request.get_response().await;
println!("UDP+TCP reply: {:?}", reply);

// The query may have a reference to the connection. Drop the query
// when it is no longer needed.
drop(request);

// Create a new TCP connections object. Pass the destination address and
// port as parameter.
let tcp_connect = TcpConnect::new(server_addr);

// A muli_stream transport connection sets up new TCP connections when
// needed.
let (tcp_conn, transport) = multi_stream::Connection::with_config(
tcp_connect,
multi_stream_config.clone(),
);

// Get a future for the run function. The run function receives
// the connection stream as a parameter.
tokio::spawn(async move {
transport.run().await;
println!("multi TCP run exited");
});

// Send a query message.
let mut request = tcp_conn.send_request(req.clone());

// Get the reply. A multi_stream connection does not have any timeout.
// Wrap get_result in a timeout.
println!("Wating for multi TCP reply");
let reply =
timeout(Duration::from_millis(500), request.get_response()).await;
println!("multi TCP reply: {:?}", reply);

drop(request);

// Some TLS boiler plate for the root certificates.
let mut root_store = RootCertStore::empty();
root_store.add_trust_anchors(webpki_roots::TLS_SERVER_ROOTS.iter().map(
|ta| {
OwnedTrustAnchor::from_subject_spki_name_constraints(
ta.subject,
ta.spki,
ta.name_constraints,
)
},
));

// TLS config
let client_config = ClientConfig::builder()
.with_safe_defaults()
.with_root_certificates(root_store)
.with_no_client_auth();

// Currently the only support TLS connections are the ones that have a
// valid certificate. Use a well known public resolver.
let google_server_addr =
SocketAddr::new(IpAddr::from_str("8.8.8.8").unwrap(), 853);

// Create a new TLS connections object. We pass the TLS config, the name of
// the remote server and the destination address and port.
let tls_connect = TlsConnect::new(
client_config,
"dns.google".try_into().unwrap(),
google_server_addr,
);

// Again create a multi_stream transport connection.
let (tls_conn, transport) = multi_stream::Connection::with_config(
tls_connect,
multi_stream_config,
);

// Start the run function.
tokio::spawn(async move {
transport.run().await;
println!("TLS run exited");
});

let mut request = tls_conn.send_request(req.clone());
println!("Wating for TLS reply");
let reply =
timeout(Duration::from_millis(500), request.get_response()).await;
println!("TLS reply: {:?}", reply);

drop(request);

// Create a transport connection for redundant connections.
let (redun, transp) = redundant::Connection::new();

// Start the run function on a separate task.
let run_fut = transp.run();
tokio::spawn(async move {
run_fut.await;
println!("redundant run terminated");
});

// Add the previously created transports.
redun.add(Box::new(udptcp_conn)).await.unwrap();
redun.add(Box::new(tcp_conn)).await.unwrap();
redun.add(Box::new(tls_conn)).await.unwrap();

// Start a few queries.
for i in 1..10 {
let mut request = redun.send_request(req.clone());
let reply = request.get_response().await;
if i == 2 {
println!("redundant connection reply: {:?}", reply);
}
}

drop(redun);

// Create a new datagram transport connection. Pass the destination address
// and port as parameter. This transport does not retry over TCP if the
// reply is truncated. This transport does not have a separate run
// function.
let udp_connect = UdpConnect::new(server_addr);
let dgram_conn =
dgram::Connection::with_config(udp_connect, dgram_config);

// Send a message.
let mut request = dgram_conn.send_request(req.clone());
//
// Get the reply
let reply = request.get_response().await;
println!("Dgram reply: {:?}", reply);

// Create a single TCP transport connection. This is usefull for a
// single request or a small burst of requests.
let tcp_conn = match TcpStream::connect(server_addr).await {
Ok(conn) => conn,
Err(err) => {
println!(
"TCP Connection to {} failed: {}, exiting",
server_addr, err
);
return;
}
};

let (tcp, transport) = stream::Connection::new(tcp_conn);
tokio::spawn(async move {
transport.run().await;
println!("single TCP run terminated");
});

// Send a request message.
let mut request = tcp.send_request(req);

// Get the reply
let reply = request.get_response().await;
println!("TCP reply: {:?}", reply);

drop(tcp);
}
2 changes: 1 addition & 1 deletion examples/readzone.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ fn main() {
start.elapsed().unwrap().as_secs_f32()
);
let mut i = 0;
while let Some(_) = zone.next_entry().unwrap() {
while zone.next_entry().unwrap().is_some() {
i += 1;
if i % 100_000_000 == 0 {
eprintln!(
Expand Down
18 changes: 18 additions & 0 deletions src/base/message.rs
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,18 @@ impl<Octs> Message<Octs> {
Ok(unsafe { Self::from_octets_unchecked(octets) })
}

/// Creates a message from octets, returning the octets if it fails.
pub fn try_from_octets(octets: Octs) -> Result<Self, Octs>
where
Octs: AsRef<[u8]>,
{
if Message::check_slice(octets.as_ref()).is_err() {
Err(octets)
} else {
Ok(unsafe { Self::from_octets_unchecked(octets) })
}
}

/// Creates a message from a bytes value without checking.
///
/// # Safety
Expand Down Expand Up @@ -1194,6 +1206,12 @@ impl From<ParseError> for CopyRecordsError {
}
}

impl From<PushError> for CopyRecordsError {
fn from(err: PushError) -> Self {
CopyRecordsError::Push(err)
}
}

//--- Display and Error

impl fmt::Display for CopyRecordsError {
Expand Down
Loading
Loading