I want to serve some static files (.js, .css, ...) from my hyper server.
Currently the only way I can think of is inlining the files as strings / load them on startup.
Is there a better way to directly serve an entire directory or selected files?
After typing in the words "hyper static" into crates.io, the first result was hyper-staticfile. The project's GitHub repository has an examples directory, with one such example:
extern crate futures;
extern crate hyper;
extern crate hyper_staticfile;
extern crate tokio_core;
// This example serves the docs from target/doc/hyper_staticfile at /doc/
//
// Run `cargo doc && cargo run --example doc_server`, then
// point your browser to http://localhost:3000/
use futures::{Future, Stream, future};
use hyper::Error;
use hyper::server::{Http, Request, Response, Service};
use hyper_staticfile::Static;
use std::path::Path;
use tokio_core::reactor::{Core, Handle};
use tokio_core::net::TcpListener;
type ResponseFuture = Box<Future<Item=Response, Error=Error>>;
struct MainService {
static_: Static,
}
impl MainService {
fn new(handle: &Handle) -> MainService {
MainService {
static_: Static::new(handle, Path::new("target/doc/")),
}
}
}
impl Service for MainService {
type Request = Request;
type Response = Response;
type Error = Error;
type Future = ResponseFuture;
fn call(&self, req: Request) -> Self::Future {
if req.path() == "/" {
let res = Response::new()
.with_status(hyper::StatusCode::MovedPermanently)
.with_header(hyper::header::Location::new("/hyper_staticfile/"));
Box::new(future::ok(res))
} else {
self.static_.call(req)
}
}
}
fn main() {
let mut core = Core::new().unwrap();
let handle = core.handle();
let addr = "127.0.0.1:3000".parse().unwrap();
let listener = TcpListener::bind(&addr, &handle).unwrap();
let http = Http::new();
let server = listener.incoming().for_each(|(sock, addr)| {
let s = MainService::new(&handle);
http.bind_connection(&handle, sock, addr, s);
Ok(())
});
println!("Doc server running on http://localhost:3000/");
core.run(server).unwrap();
}
Related
I've viewed so many stack overflows, but can't get my head around how to fix this.
Here are my deps:
[dependencies]
mongodb = { version = "2.3.1"}
tokio = "1.22.0"
twilio-openapi = "1.0.0"
async-trait = "0.1.58"
futures = "0.3.25"
I've created this async_trait repository trait for implementing various data access repositories:
use std::pin::Pin;
use async_trait::async_trait;
use futures::Future;
use mongodb::{bson::Document, error::Error, options::FindOneOptions, Client};
pub mod notification_settings;
#[async_trait]
pub trait Repository<'a> {
fn with_client(client: &'a mut &Client) -> Self;
async fn find_one<'b>(
self,
filter: Document,
find_one_options: FindOneOptions,
) -> Pin<Box<(dyn Future<Output = Result<Option<Document>, Error>> + Send + 'b)>>;
}
It seems like async_trait sort of requires returning a Pin<Box<dyn Future<Output = X> + Send + 'b)>> Where you can specify a certain lifetime on the borrow. I'm not sure exactly why or if there are ways around this, but again still learning this.
And then here is my struct implementing the trait:
use std::{marker::Send, pin::Pin};
use async_trait::async_trait;
use futures::Future;
use mongodb::{bson::Document, error::Error, options::FindOneOptions, Client};
pub struct NotificationSettingsRepository<'a> {
pub(crate) client: &'a Client,
}
#[async_trait]
impl<'a> super::Repository<'a> for NotificationSettingsRepository<'a> {
fn with_client(client: &'a mut &Client) -> Self {
NotificationSettingsRepository { client: client }
}
async fn find_one<'b>(
self,
filter: Document,
find_one_options: FindOneOptions,
) -> Pin<Box<(dyn Future<Output = Result<Option<Document>, Error>> + Send + 'b)>> {
let collection = self
.client
.database("Occasionally")
.collection::<Document>("NotificationSettings");
let document = collection.find_one(filter, find_one_options);
Box::pin(async { document.await })
}
}
Problem is on collection.find_one(filter, find_one_options); I get an error about:
`collection` does not live long enough
borrowed value does not live long enoughrustcClick for full compiler diagnostic
notification_settings.rs(30, 5): `collection` dropped here while still borrowed
notification_settings.rs(17, 23): lifetime `'b` defined here
notification_settings.rs(21, 10): type annotation requires that `collection` is borrowed for `'b`
This makes some sense because the definition of find_one looks like this:
find_one(&self, filter: impl Into<Option<Document>>, options: impl Into<Option<FindOneOptions>>) -> Result<Option<T>>
So it is borrowing the collection via &self
I've tried a few things like adding an async { } wrapper on the Future of find_one. But I'm not sure what I'm doing there. Please help and thank you!
I just want to get a JSON from the following URL.
So I used this code:
extern crate reqwest;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let res = reqwest::Client::new()
.get("https://api.github.com/users/octocat")
.send()?
.text()?;
println!("{}", res);
Ok(())
}
But I don't know how to solve the error :
error[E0277]: the `?` operator can only be applied to values that implement `std::ops::Try`
--> src\main.rs:19:15
|
19 | let res = reqwest::Client::new()
| _______________^
20 | | .get("https://api.github.com/users/octocat")
21 | | .send()?
| |________________^ the `?` operator cannot be applied to type `impl std::future::Future`
|
= help: the trait `std::ops::Try` is not implemented for `impl std::future::Future`
= note: required by `std::ops::Try::into_result`
error: aborting due to previous error
For more information about this error, try `rustc --explain E0277`.
error: could not compile `Desktop`.
but I can also obtain what I want with a simple
curl https://api.github.com/users/octocat
I've tried to add use std::ops::Try; but it doesn't work better.
The reqwest crate uprovides an asynchronous api by default. Therefore you have to .await before handling the error with the ? operator. You also have to use an async runtime, such as tokio:
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let resp = reqwest::Client::new()
.get("https://api.github.com/users/octocat")
.send()
.await?
.json::<std::collections::HashMap<String, String>>()
.await?;
println!("{:#?}", resp);
Ok(())
}
Note that to use convert the response to json as shown above, you must enable the json feature in your Cargo.toml:
reqwest = { version = "0.10.8", features = ["json"] }
If you don't want to use an async runtime, you can enable the blocking reqwest client:
[dependencies]
reqwest = { version = "0.10", features = ["blocking", "json"] }
And use it like so:
fn main() -> Result<(), Box<dyn std::error::Error>> {
let resp = reqwest::blocking::Client::new()
.get("https://api.github.com/users/octocat")
.send()?
.json::<std::collections::HashMap<String, String>>()?;
println!("{:#?}", resp);
Ok(())
}
Github's api requires a couple other config options. Here is a minimal working example with reqwest and the github api:
use reqwest::header::{HeaderMap, HeaderValue, USER_AGENT};
use serde::{Deserialize};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let mut headers = HeaderMap::new();
// add the user-agent header required by github
headers.insert(USER_AGENT, HeaderValue::from_static("reqwest"));
let resp = reqwest::blocking::Client::new()
.get("https://api.github.com/users/octocat")
.headers(headers)
.send()?
.json::<GithubUser>()?;
println!("{:#?}", resp);
Ok(())
}
// Note that there are many other fields
// that are not included for this example
#[derive(Deserialize, Debug)]
pub struct GithubUser {
login: String,
id: usize,
url: String,
#[serde(rename = "type")]
ty: String,
name: String,
followers: usize
}
I would like to download a large file (hundreds of megabytes) with the AsyncHTTPClient library based on SwiftNIO. I would like this file to be streamed to the filesystem, while consuming as little RAM as possible (ideally it shouldn't keep the whole file in the RAM), and also being able to report the download progress with a simple print output that shows the completion percentage.
As far as I understand, I need to implement an HTTPClientResponseDelegate, but what exact API should I use for file writes? Can file writes be blocking, while still allowing the HTTP client to progress? How would the delegate code look in this scenario?
Turns out, HTTPClientResponseDelegate allows returning a future in its functions to allow it to correctly handle backpressure. Combined this approach with NonBlockingFileIO and NIOFileHandle, the delegate that writes the file to disk with progress reporting while it's downloaded looks like this:
import AsyncHTTPClient
import NIO
import NIOHTTP1
final class FileDownloadDelegate: HTTPClientResponseDelegate {
typealias Response = (totalBytes: Int?, receivedBytes: Int)
private var totalBytes: Int?
private var receivedBytes = 0
private let handle: NIOFileHandle
private let io: NonBlockingFileIO
private let reportProgress: (_ totalBytes: Int?, _ receivedBytes: Int) -> ()
private var writeFuture: EventLoopFuture<()>?
init(
path: String,
reportProgress: #escaping (_ totalBytes: Int?, _ receivedBytes: Int) -> ()
) throws {
handle = try NIOFileHandle(path: path, mode: .write, flags: .allowFileCreation())
let pool = NIOThreadPool(numberOfThreads: 1)
pool.start()
io = NonBlockingFileIO(threadPool: pool)
self.reportProgress = reportProgress
}
func didReceiveHead(
task: HTTPClient.Task<Response>,
_ head: HTTPResponseHead
) -> EventLoopFuture<()> {
if let totalBytesString = head.headers.first(name: "Content-Length"),
let totalBytes = Int(totalBytesString) {
self.totalBytes = totalBytes
}
return task.eventLoop.makeSucceededFuture(())
}
func didReceiveBodyPart(
task: HTTPClient.Task<Response>,
_ buffer: ByteBuffer
) -> EventLoopFuture<()> {
receivedBytes += buffer.readableBytes
reportProgress(totalBytes, receivedBytes)
let writeFuture = io.write(fileHandle: handle, buffer: buffer, eventLoop: task.eventLoop)
self.writeFuture = writeFuture
return writeFuture
}
func didFinishRequest(task: HTTPClient.Task<Response>) throws -> Response {
writeFuture?.whenComplete { [weak self] _ in
try? self?.handle.close()
self?.writeFuture = nil
}
return (totalBytes, receivedBytes)
}
}
With this code, the process downloading and writing the file does not consume more than 5MB of RAM for a ~600MB downloaded file.
I am trying to have a struct that starts an event loop, listens for TCP connections and calls a callback for each connection.
(The callback will be handed some prepossessed data from the socket. In my example below I just hand it the IP address of the connection but in my real code I will parse the contents that I receive with serde into a struct and pass that into the callback. I hope that doesn't invalidate the following "not working example").
My Cargo.toml:
[package]
name = "lifetime-problem"
version = "0.1.0"
edition = "2018"
[dependencies]
tokio-tcp = "0.1.3"
tokio = "0.1.14"
[[bin]]
name = "lifetime-problem"
path = "main.rs"
and main.rs:
use tokio::prelude::*;
struct Test {
printer: Option<Box<Fn(std::net::SocketAddr) + Sync>>,
}
impl Test {
pub fn start(&mut self) -> Result<(), Box<std::error::Error>> {
let addr = "127.0.0.1:4242".parse::<std::net::SocketAddr>()?;
let listener = tokio::net::TcpListener::bind(&addr)?;
let server = listener
.incoming()
.map_err(|e| eprintln!("failed to accept socket; error = {:?}", e))
.for_each(move |socket: tokio::net::TcpStream| {
let address = socket.peer_addr().expect("");
match self.printer {
Some(callback) => { callback(address); }
None => { println!("{}", address); }
}
Ok(())
});
tokio::run(server);
Ok(())
}
}
fn main() {
let mut x = Test{ printer: None };
x.start();
}
I have tried several things starting from this code (which is adopted directly from the Tokio example).
If I use the code like posted above I get:
error[E0277]: (dyn std::ops::Fn(std::net::SocketAddr) + std::marker::Sync + 'static) cannot be sent between threads safely
for the line 24 (tokio::run(server)).
If I add the Send trait on the Fn in the printer field XOR if I remove the move in the closure in the for_each call I get another error instead:
error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
which points me to the closure that apparently cannot outlive the start method where it is defined but tokio::run seems to have conflicting requirements for it.
Do you know if I am addressing the callback pattern in totally the wrong way or if there is just some minor error in my code?
First things first:
Compiler will translate Box<Fn(std::net::SocketAddr) + Sync> to Box<Fn(std::net::SocketAddr) + Sync + 'static> unless the lifetime is explicitly specified.
Let's have a look at the errors:
error[E0277]: (dyn std::ops::Fn(std::net::SocketAddr) + std::marker::Sync + 'static) cannot be sent between threads safely
This is self-explanatory. You are trying to move &mut T to another thread, but cannot, because T here is not Send. To send &mut T to another thread T too needs to be of type Send.
Here is the minimal code that will give the same error:
use std::fmt::Debug;
fn func<T> (i:&'static mut T) where T: Debug {
std::thread::spawn(move || {
println!("{:?}", i);
});
}
If I make T above to also be of type Send, the error goes away.
But in your case when you add the Send trait, it gives lifetime error. Why?
&mut self has some lifetime greater than the function start() set by the caller, but there's no guarantee that its 'static. You move this reference into the closure which is passed to the thread and can potentially outlive the scope it is closing over, leading to a dangling reference.
Here's a minimal version, that would give the same error.
use std::fmt::Debug;
fn func<'a, T:'a> (i:&'a mut T) where T: Debug + Sync + Send {
std::thread::spawn(move || {
println!("{:?}", i);
});
}
Sync is not really required here as it is &mut T. Changing &mut T to &T (retaining Sync), will also result into the same error. The onus here is on references and not mutability. So you see, there is some lifetime 'a and it is moved into a closure (given to a thread), which means the closure now contains a reference (disjoint from the main context). So now, what is 'a and how long will it live from the closure's perspective that is invoked from another thread? Not inferable! As a result, the compiler complains saying cannot infer an appropriate lifetime due to conflicting requirements.
If we tweak the code a bit to;
impl Test {
pub fn start(&'static mut self) -> Result<(), Box<std::error::Error>> {
let addr = "127.0.0.1:4242".parse::<std::net::SocketAddr>()?;
let listener = tokio::net::TcpListener::bind(&addr)?;
let server = listener
.incoming()
.map_err(|e| eprintln!("failed to accept socket; error = {:?}", e))
.for_each(move |socket: tokio::net::TcpStream| {
let address = socket.peer_addr().expect("");
match &self.printer {
Some(callback) => { callback(address); }
None => { println!("{}", address); }
}
Ok(())
});
tokio::run(server);
Ok(())
}
}
it will compile fine. There's a guarantee there that self has a 'static lifetime. Please note that in the match statement we need to pass &self.printer, as you cannot move out of a borrowed context.
However, this expects Test to be declared static and that too a mutable one, which is generally not the best way, if you have other options.
Another way is; if it's ok for you to pass Test by value to start() and then further move it into for_each(), the code would look like this:
use tokio::prelude::*;
struct Test {
printer: Option<Box<Fn(std::net::SocketAddr) + Send>>,
}
impl Test {
pub fn start(mut self) -> Result<(), Box<std::error::Error>> {
let addr = "127.0.0.1:4242".parse::<std::net::SocketAddr>()?;
let listener = tokio::net::TcpListener::bind(&addr)?;
let server = listener
.incoming()
.map_err(|e| eprintln!("failed to accept socket; error = {:?}", e))
.for_each(move |socket: tokio::net::TcpStream| {
let address = socket.peer_addr().expect("");
match &self.printer {
Some(callback) => {
callback(address);
}
None => {
println!("{}", address);
}
}
Ok(())
});
tokio::run(server);
Ok(())
}
}
fn main() {
let mut x = Test { printer: None };
x.start();
}
I have a test that opens and listens to a Unix Domain Socket. The socket is opened and reads data without issues, but it doesn't shutdown gracefully.
This is the error I get when I try to run the test a second time:
thread 'test_1' panicked at 'called Result::unwrap() on an Err
value: Error { repr: Os { code: 48, message: "Address already in use"
} }', ../src/libcore/result.rs:799 note: Run with RUST_BACKTRACE=1
for a backtrace.
The code is available at the Rust playground and there's a Github Gist for it.
use std::io::prelude::*;
use std::thread;
use std::net::Shutdown;
use std::os::unix::net::{UnixStream, UnixListener};
Test Case:
#[test]
fn test_1() {
driver();
assert_eq!("1", "2");
}
Main entry point function
fn driver() {
let listener = UnixListener::bind("/tmp/my_socket.sock").unwrap();
thread::spawn(|| socket_server(listener));
// send a message
busy_work(3);
// try to disconnect the socket
let drop_stream = UnixStream::connect("/tmp/my_socket.sock").unwrap();
let _ = drop_stream.shutdown(Shutdown::Both);
}
Function to send data in intervals
#[allow(unused_variables)]
fn busy_work(threads: i32) {
// Make a vector to hold the children which are spawned.
let mut children = vec![];
for i in 0..threads {
// Spin up another thread
children.push(thread::spawn(|| socket_client()));
}
for child in children {
// Wait for the thread to finish. Returns a result.
let _ = child.join();
}
}
fn socket_client() {
let mut stream = UnixStream::connect("/tmp/my_socket.sock").unwrap();
stream.write_all(b"hello world").unwrap();
}
Function to handle data
fn handle_client(mut stream: UnixStream) {
let mut response = String::new();
stream.read_to_string(&mut response).unwrap();
println!("got response: {:?}", response);
}
Server socket that listens to incoming messages
#[allow(unused_variables)]
fn socket_server(listener: UnixListener) {
// accept connections and process them, spawning a new thread for each one
for stream in listener.incoming() {
match stream {
Ok(mut stream) => {
/* connection succeeded */
let mut response = String::new();
stream.read_to_string(&mut response).unwrap();
if response.is_empty() {
break;
} else {
thread::spawn(|| handle_client(stream));
}
}
Err(err) => {
/* connection failed */
break;
}
}
}
println!("Breaking out of socket_server()");
drop(listener);
}
Please learn to create a minimal reproducible example and then take the time to do so. In this case, there's no need for threads or functions or testing frameworks; running this entire program twice reproduces the error:
use std::os::unix::net::UnixListener;
fn main() {
UnixListener::bind("/tmp/my_socket.sock").unwrap();
}
If you look at the filesystem before and after the test, you will see that the file /tmp/my_socket.sock is not present before the first run and it is present before the second run. Deleting the file allows the program to run to completion again (at which point it recreates the file).
This issue is not unique to Rust:
Note that, once created, this socket file will continue to exist, even after the server exits. If the server subsequently restarts, the file prevents re-binding:
[...]
So, servers should unlink the socket pathname prior to binding it.
You could choose to add some wrapper around the socket that would automatically delete it when it is dropped or create a temporary directory that is cleaned when it is dropped, but I'm not sure how well that would work. You could also create a wrapper function that deletes the file before it opens the socket.
Unlinking the socket when it's dropped
use std::path::{Path, PathBuf};
struct DeleteOnDrop {
path: PathBuf,
listener: UnixListener,
}
impl DeleteOnDrop {
fn bind(path: impl AsRef<Path>) -> std::io::Result<Self> {
let path = path.as_ref().to_owned();
UnixListener::bind(&path).map(|listener| DeleteOnDrop { path, listener })
}
}
impl Drop for DeleteOnDrop {
fn drop(&mut self) {
// There's no way to return a useful error here
let _ = std::fs::remove_file(&self.path).unwrap();
}
}
You may also want to consider implementing Deref / DerefMut to make this into a smart pointer for sockets:
impl std::ops::Deref for DeleteOnDrop {
type Target = UnixListener;
fn deref(&self) -> &Self::Target {
&self.listener
}
}
impl std::ops::DerefMut for DeleteOnDrop {
fn deref_mut(&mut self) -> &mut Self::Target {
&mut self.listener
}
}
Unlinking the socket before it's opened
This is much simpler:
use std::path::Path;
fn bind(path: impl AsRef<Path>) -> std::io::Result<UnixListener> {
let path = path.as_ref();
std::fs::remove_file(path)?;
UnixListener::bind(path)
}
Note that you can combine the two solutions, such that the socket is deleted before creation and when it's dropped.
I think that deleting during creation is a less-optimal solution: if you ever start a second server, you'll prevent the first server from receiving any more connections. It's probably better to error and tell the user instead.