Simpler alternative to TempFile in multipart form file upload (Like using Vec<u8> directly)

[Alvenix]

For file upload, I have been using TempFile. However, I don't feel that it is very ergonomic. You have to store the file first to read it and later delete it.

Here is some pseudo code which showcase how sometimes I had to use it.

#[post("/upload", data = "<file>")]
async fn process_file(mut file: TempFile<'_>) -> std::io::Result<()> {
    let uuid = Uuid::new_v4();
    let random_file_name = format!("/tmp/{uuid}");
    file.persist_to(&random_file_name).await?;
  
    let content = fs::read(&random_file_name)?;

    fs::delete(&random_file_name)?;

    Ok(())
}

I did not find existing alternative in Rocket.

[SergioBenitez]

Well, what are you trying to do with the data? A TempFile is to be used when you're trying to save the data to disk, or when the amount of data might exceed what you can store in memory. Given that you read the entire contents of the file into memory, your code is either wrong, you're accepting the crash due to memory exhaustion, or you're properly limiting file size.

If you're doing the latter, then indeed this is inefficient. Unfortunately Rocket doesn't and can't have Vec<u8> mean "a byte buffer" because Vec<u8> already means "a sequence of 8-bit unsigned integers," as far as forms go. We could add a Bytes or Buffer type or similar with the intended meaning, however.

If you're doing some sort of cheap computation with the data, then you probably don't want to buffer it all beyond, say, 4k streaming chunks, and store only the computed value. This is relatively easy to implement in Rocket via the FromFormField trait and gives you complete control over data limiting, buffering, and streaming.

[Alvenix]

Well, what are you trying to do with the data? A TempFile is to be used when you're trying to save the data to disk, or when the amount of data might exceed what you can store in memory. Given that you read the entire contents of the file into memory, your code is either wrong, you're accepting the crash due to memory exhaustion, or you're properly limiting file size.

If you're doing the latter, then indeed this is inefficient. Unfortunately Rocket doesn't and can't have Vec<u8> mean "a byte buffer" because Vec<u8> already means "a sequence of 8-bit unsigned integers," as far as forms go. We could add a Bytes or Buffer type or similar with the intended meaning, however.

If you're doing some sort of cheap computation with the data, then you probably don't want to buffer it all beyond, say, 4k streaming chunks, and store only the computed value. This is relatively easy to implement in Rocket via the FromFormField trait and gives you complete control over data limiting, buffering, and streaming.

Yes I limit file sizes when configuring rocket.

A Bytes or Buffer type would be nice to have instead of manually implementing FromFormField. Sometimes you need the full file (small file) in memory.

Alternatively, I think it would be better (well more ergonomic) if TempFile or some alternative struct is guaranteed to be already in a file so that path() always return a file path. Or It could have some read methods which work whether it is buffered or in file?

[Asone]

i'll profit this issue to ask if it is possible to allow creation of the TempFile instance outside the data instance ?

Some kind of multipart requests can allow multiple files in the multipart/form-data content, like how it's done in many graphQL clients using this unofficial but commonly used spec.

The current TempFile processing sticks to the idea the provided body data is the file while in the previously mentionned case the file is one of the boundaries of the body data, not the whole one.

I've wrote some kind of prototype to handle the above mentioned case with the spec, however i had to build a custom - and simplified - TempFile struct, but i must admit that my knowledge of how to deal properly with buffered data is still quite limited and i would feel way more comfortable on being able to delegate this task to the framework.

Do you think that having a method that would allow to build up TempFiles detached from the data body would be feasible ?

[SergioBenitez]

@Asone That's just a multipart form, which Rocket already handles natively. Just use TempFile as a field of a struct that implements FromForm.

[JohnDowson]

Okay, so here's my usecase. I want to hash a tempfile (preferably without writing it to disk at all), look up the hash in a database and finally persist the file if it's hash wasn't found.
Right now I have to

1. Write file to disk
2. Read it back
3. Hash it
4. Move it to a permanent location or delete

This feels unnecessarily complicated.

[Asone]

@Asone That's just a multipart form, which Rocket already handles natively. Just use TempFile as a field of a struct that implements FromForm.

The spec is a bit peculiar in my opinion as there is no way to know in advance all the fields that will be provided aside of the operations and map field.

The submitted form could be :

{
 operations : <Json Object>,
 map: <Json Object>,
 0: binary,
 1: binary
}

as

{
 operations : <Json Object>,
 map: <Json Object>,
 a: binary,
 b: binary 
 c : binary,
 0 : binary,
 anything: binary
}

This seems incompatible with the fromForm, which is why i wrote the previously mentioned prototype, as it processes all the boundaries provided acknowledging that we cant know how many there'll be in advance and what there key name will be.

But maybe i'm missing some subtleties with rust structs that would make it feasible or compatible with fromForm ? 🤷 If so, i haven't found yet :/

[JohnDowson]

This feels unnecessarily complicated.

Not to mention that it doesn't work: persisting or copying for some reason silently fails, producing no file or error.

[SergioBenitez]

FromForm can handle any arbitrary structure at all. You never need to implement your own multipart parser. For instance, the HashMap type can collect all fields of a given type, irrespective of their field name.

I suggest a thorough reading of the FromForm and Form docs.

[marziply]

We could add a Bytes or Buffer type or similar with the intended meaning, however.

I think this makes a lot of sense. I'm in agreement with other folks here that it would be advantageous to have access to that multipart data as a buffer of some kind. It's useful in some cases to store that data in memory rather than saving a temporary file, similar to Node.js Buffers, which can be used to store the multipart data in memory.

I'll have a spin at an implementation of this over the weekend, any pointers/guidance on how this might be done would be helpful since it would be my first contribution to this project.

[eric8607242]

Hello!
Is there any method to keep the file in the multipart into the memory directly without dumping the file into the tempfile now?

Very thanks for any response!