Enum Variants to Numpy Dtype

[aaronclong]

I know there is the Element trait, but it isn't easy to implement and almost impossible for Enums.

I wonder if it would be possible to introduce a new trait that would allow for conversion.

My use case was to convert avro records to numpy arrays. The exact schema for the avro file might not be known a head of time. Unfortunately, I don't think it is possible with the library's current infrastructure.

Side note, new to Rust, there may be many uninformed/unintelligent things said above.

[adamreichold]

I know there is the Element trait, but it isn't easy to implement and almost impossible for Enums.

The trait needs to capture that NumPy itself is actually able to store only primitive types or objects on the Python heap, i.e. Py<SomePyClass>, directly inside arrays. (These bindings do not support structured arrays yet, c.f. #186 , but they would still not enable storage of arbitrary Rust types.)

I wonder if it would be possible to introduce a new trait that would allow for conversion.

I think it would be helpful to look at the more of the code in question here. What data structure do you start out from and what kind of NumPy arrays do you want to create?

As a general response, type conversion in Rust are usually handled via From or TryFrom if fallible, but that would probably have to be applied on the level of the container you start out with.

Side note, new to Rust, there may be many uninformed/unintelligent things said above.

Don't worry. We have all started there.

[aaronclong]

Hi @adamreichold, the initial type is the avro value type here.
I know I will have to map it at some point to another type.

The hard part about constructing numpy array is that the type info is a range of possible types and not know until runtime/schema load.

I know the numpy arrays require primitives, but I am struggling to find a way to auto convert from some type to it's primitive.

[adamreichold]

Without more context, I guess you would end up with an enum containing different NumPy arrays, e.g.

enum ArrayOfValue<D> {
    Null,
    Boolean(PyArray<bool, D>),
    Int(PyArray<i32, D>),
}

I don't think there is a single PyArray<T, D> where T will handle this at runtime except for boxing everything up on the Python heap, e.g.

#[pyclass]
struct PyAvroValue {
  value: avro_rs::types::Value
}

and using

PyArray<Py<PyAvroValue>, D>

as your array type. But that would also require you use the unreleased code from the main branch here and I am not sure what any downstream code could do with those arrays.

[aaronclong]

@adamreichold so I want to convert those raw values into numpy arrays and not python arrays.

The ultimate output of the process would be a dataframe.

So I am trying to create arrays from those possible datatypes in Avro.

[adamreichold]

so I want to convert those raw values into numpy arrays and not python arrays.

PyArray is the Rust-side representation of a NumPy array provided by this crate.

[aaronclong]

so I want to convert those raw values into numpy arrays and not python arrays.

PyArray is the Rust-side representation of a NumPy array provided by this crate.

Totally missed that in the documentation. Will review that again, and try using that struct.

[aldanor]

Is there still anything unresolved in this issue?

[adamreichold]

Is there still anything unresolved in this issue?

I don't think so. The op has not reported in again but we can always re-open if there are still open questions.

[aaronclong]

Sorry, I have been on vacation. I still can't get this work despite my best efforts.

I don't know what I am doing wrong. It is mostly like something in my understanding. However, I still don't know how to go from enum variants to PyArray.

[aldanor]

I think it would help if, putting Rust aside for a moment, you explained what it's the result you wanted to see and whether it's implementable in pure Python as it stands?

If the ultimate result "is the dataframe" as you mentioned, what you're looking for is probably structured arrays, no? These numpy arrays can be converted directly to pd.DataFrame.

If that's the case, then what you need is #186 to be implemented which, coincidentally, is something that we're working on right now (in #254).

[aaronclong]

@aldanor I have been following that thread as well, and from the context, it seems to be a solution. Some of it goes over my head, so I can't be entirely certain.

What I want to do can be done in pure Python, but I am trying to improve the performance for large datasets. It is basically load large dataframes from Avro files.

[adamreichold]

it seems to be a solution.

I would not necessarily wait until this all pans out. Depending on how generic you data loading code needs to be, you might just get away with creating a single array of primitives for each column which would work with what is already released on crates.io and also be converted into a Pandas DataFrame, e.g. from a dictionary mapping column name to column array.

Finally, again depending on the actual task that is trying to be accomplished, a Rust-level data frame library like Polars might to do the trick, assuming you do not necessarily need to tap into the Pandas ecosystem to leverage an existing code base.

[aldanor]

Yes, the simplest solution for now, with today's rust-numpy, is probably creating a named dict of 1-D scalar numpy arrays. You can then pass that into pd.DataFrame constructor and it should create a new dataframe instantly without copying.