[API Proposal]: Future of Numerics and AI - Provide a downlevel `System.Numerics.Tensors.TensorPrimitives` class

[tannergooding]

Future of Numerics and AI

.NET provides a broad range of support for various development domains, ranging from the creation of performance-oriented framework code to the rapid development of cloud native services, and beyond. In recent years, especially with the rise of AI and machine learning, there has been a prevalent push towards improving numerical support and allowing developers to more easily write and consume general purpose and reusable algorithms that work with a range of types and scenarios. While .NET's support for scalar algorithms and fixed-sized vectors/matrices is strong and continues to grow, its built-in library support for other concepts, such as tensors and arbitrary length vectors/matrices, is behind the times. Developers writing .NET applications, services, and libraries currently need to seek external dependencies in order to utilize functionality that is considered core or built-in to other ecosystems. In particular, for developers incorporating AI and copilots into their existing .NET applications and services, we strive to ensure that the core numerics support necessary to be successful is available and efficient, and that .NET developers are not forced to seek out non-.NET solutions in order for their .NET projects to be successful.

.NET Framework Compatible APIs

While there are many reasons for developers to migrate to modern .NET and it is the goto target for many new libraries or applications, there remains many large repos that have been around for years where migration can be non-trivial. And while there are many new language and runtime features which cannot ever work on .NET Framework, there still remains a subset of APIs that would be beneficial to provide and which can help those existing codebases bridge the gap until they are able to successfully complete migration.

It is therefore proposed that an out of band System.Numerics.Tensors package be provided which provides a core set of APIs that are compatible with and appropriate for use on .NET Standard 2.0. There are also two types that would be beneficial to polyfill downlevel as part of this process, System.Half and System.MathF, which will significantly improves the usability of the libraries for common scenarios.

Provide the Tensor class

Note

The following is an extreme simplification meant to give the general premise behind the APIs being exposed. It is intentionally skimming over several concepts and deeper domain specific terminology that is not critical to cover for the design proposal to be reviewed.

At a very high overview, you have scalars, vectors, and matrices. A scalar is largely just a single value T, a vector is a set of scalars, a matrix is a set of vectors. All of these can be loosely thought of as types of tensors and they have various relationships and can be used to build up and represent higher types. That is, you could consider that T is a scalar, or a vector of length 1, or a 1x1 matrix. Likewise, you can then consider that T[] is a vector or unknown length and that T[,] represents a 2-dimensional matrix and so on. All of these can be defined as types of tensors. -- As a note, this terminology is partially where the name for std::vector<T> in C++ (and similar in other languages) comes from. This is also where the general considerations of "vectorization" when writing SIMD or SIMT arise, although they are typically over fixed-length, rather than arbitrary length.

From the mathematical perspective, many of the operations you can do on scalars also apply to the higher types. There can sometimes be limitations or other considerations that can specialize or restrict these operations, but in principle they tend to exist and work mostly the same way. It is therefore generally desirable to support many such operations more broadly. This is particularly prevalent for "element-wise" operations which could be thought of as effectively doing array.Select((x) => M(x)) and where an explicit API can often provide significant performance improvements over the naive approach.

The core set of APIs described below cover the main arithmetic and conversion operators provided for T as well as element-wise operations for the functionality exposed by System.Math/System.MathF. Some design notes include:

• Where the scalar API Method would return a bool, the vector API has two overloads MethodAny and MethodAll.
• APIS will require that developers utilize Span<T> to access these APIs, overloads taking T[] are not provided
• APIs will support the destination buffer and one of the source buffers being the same.
• APIs taking more than 1 operand will support the other operands being a scalar and it indicating the same value is used for every element.
• In order to make usage of the APIs less "clunky", particularly given these are mathematical APIs where building up complex expressions may be more prevalent, it is proposed a slight deviation from the normal API signature is taken. Rather than simply returning the number of elements written to the destination span, it is proposed that the APIs return a Span<T> instead. This would simply be destination sliced to the appropriate length. This still provides the required information on number of elements written, but gives the additional advantage that the result can be immediately passed into the next user of the algorithm without requiring the user to slice or do length checks themselves.

For targeting modern .NET, there will be a separate future proposal detailing a Tensor<T> type. This then matches a similar split we have for other generic types such as Vector<T> and Vector. The non-generic Tensor class holds extension methods, APIs that are only meant to support a particular set of T, and static APIs. While Tensor<T> will hold operators, instance methods, and core properties. The APIs defined for use on .NET Framework are effectively the "workhorse" APIs that Tensor<T> would then delegate to. They more closely resemble the signatures from the BLAS and LAPACK libraries, which are the industry standard baseline for Linear Algebra operations and allow tensor like functionality to be supported for arbitrary memory while allowing modern .NET to provide a type safe and friendlier way to work with such functionality that can simultaneously take advantage of newer language/runtime features, such as static virtuals in interfaces or generic math.

namespace System.Numerics.Tensors;

public static partial class Tensor
{
    // Element-wise Arithmetic

    public static Span<float> Add(ReadOnlySpan<float> left, ReadOnlySpan<float> right, Span<float> destination);
    public static Span<float> Add(ReadOnlySpan<float> left, T right, Span<float> destination);

    public static Span<float> Subtract(ReadOnlySpan<float> left, ReadOnlySpan<float> right, Span<float> destination);
    public static Span<float> Subtract(ReadOnlySpan<float> left, T right, Span<float> destination);

    public static Span<float> Multiply(ReadOnlySpan<float> left, ReadOnlySpan<float> right, Span<float> destination);
    public static Span<float> Multiply(ReadOnlySpan<float> left, T right, Span<float> destination); // BLAS1: scal

    public static Span<float> Divide(ReadOnlySpan<float> left, ReadOnlySpan<float> right, Span<float> destination);
    public static Span<float> Divide(ReadOnlySpan<float> left, T right, Span<float> destination);

    public static Span<float> Negate(ReadOnlySpan<float> values, Span<float> destination);

    public static Span<float> AddMultiply(ReadOnlySpan<float> left, ReadOnlySpan<float> right, ReadOnlySpan<float> multiplier, Span<float> destination);
    public static Span<float> AddMultiply(ReadOnlySpan<float> left, ReadOnlySpan<float> right, T multiplier, Span<float> destination);
    public static Span<float> AddMultiply(ReadOnlySpan<float> left, T right, ReadOnlySpan<float> multiplier, Span<float> destination);

    public static Span<float> MultiplyAdd(ReadOnlySpan<float> left, ReadOnlySpan<float> right, ReadOnlySpan<float> addend, Span<float> destination);
    public static Span<float> MultiplyAdd(ReadOnlySpan<float> left, ReadOnlySpan<float> right, T addend, Span<float> destination); // BLAS1: axpy
    public static Span<float> MultiplyAdd(ReadOnlySpan<float> left, T right, ReadOnlySpan<float> addend, Span<float> destination);

    public static Span<float> Exp(ReadOnlySpan<float> x, Span<float> destination);
    public static Span<float> Log(ReadOnlySpan<float> x, Span<float> destination);

    public static Span<float> Cosh(ReadOnlySpan<float> x, Span<float> destination);
    public static Span<float> Sinh(ReadOnlySpan<float> x, Span<float> destination);
    public static Span<float> Tanh(ReadOnlySpan<float> x, Span<float> destination);

    // Vector Arithmetic

    // A measure of similarity between two non-zero vectors of an inner product space.
    // It is widely used in natural language processing and information retrieval.
    public static float CosineSimilarity(ReadOnlySpan<float> x, ReadOnlySpan<float> y);

    // A measure of distance between two points in a Euclidean space.
    // It is widely used in mathematics, engineering, and machine learning.
    public static float Distance(ReadOnlySpan<float> x, ReadOnlySpan<float> y);

    // A mathematical operation that takes two vectors and returns a scalar.
    // It is widely used in linear algebra and machine learning.
    public static float Dot(ReadOnlySpan<float> x, ReadOnlySpan<float> y); // BLAS1: dot

    // A mathematical operation that takes a vector and returns a unit vector in the same direction.
    // It is widely used in linear algebra and machine learning.
    public static float Normalize(ReadOnlySpan<float> x); // BLAS1: nrm2

    // A function that takes a collection of real numbers and returns a probability distribution.
    // It is widely used in machine learning and deep learning.
    public static float SoftMax(ReadOnlySpan<float> x);

    // A function that takes a real number and returns a value between 0 and 1.
    // It is widely used in machine learning and deep learning.
    public static Span<float> Sigmoid(ReadOnlySpan<float> x, Span<float> destination);

    public static float Max(ReadOnlySpan<float> value);
    public static float Min(ReadOnlySpan<float> value);

    public static int IndexOfMaxMagnitude(ReadOnlySpan<float> value); // BLAS1: iamax
    public static int IndexOfMinMagnitude(ReadOnlySpan<float> value);

    public static float Sum(ReadOnlySpan<float> value);
    public static float SumOfSquares(ReadOnlySpan<float> value);
    public static float SumOfMagnitudes(ReadOnlySpan<float> value); // BLAS1: asum

    public static float Product(ReadOnlySpan<float> value);
    public static float ProductOfSums(ReadOnlySpan<float> left, ReadOnlySpan<float> right);
    public static float ProductOfDifferences(ReadOnlySpan<float> left, ReadOnlySpan<float> right);

    // Vector Conversions

    public static Span<Half> ConvertToHalf(ReadOnlySpan<float> value, Span<Half> destination);
    public static Span<float> ConvertToSingle(ReadOnlySpan<Half> value, Span<float> destination);
}

Polyfill System.Half in Microsoft.Bcl.Half

System.Half is a core interchange type for AI scenarios, often being used to minify the storage impact for the tens of thousands to millions of data points that need to be stored. It is not, however, as frequently used for computation.

We initially exposed this type in .NET 5 purely as an interchange type and it is therefore lacking the arithmetic operators. These operators were later added in .NET 6/7 as a part of the generic math initiative. For .NET Framework, the interchange surface area should be sufficient and will follow the general guidance required for polyfills that they meet the initial shape we shipped, even if the version it shipped on is no longer in support (that is, the .NET Standard 2.0 surface area needs to remain compatible with .NET 5, even though .NET 5 is out of support).

namespace System;

public readonly struct Half : IComparable, IComparable<Half>, IEquatable<Half>, IFormattable
{
    public static Half Epsilon { get; }
    public static Half MaxValue { get; }
    public static Half MinValue { get; }
    public static Half NaN { get; }
    public static Half NegativeInfinity { get; }
    public static Half PositiveInfinity { get; }

    public static bool operator ==(Half left, Half right);
    public static bool operator !=(Half left, Half right);

    public static bool operator >(Half left, Half right);
    public static bool operator >=(Half left, Half right);

    public static bool operator <(Half left, Half right);
    public static bool operator <=(Half left, Half right);

    public static explicit operator double(Half value);
    public static explicit operator Half(float value);
    public static explicit operator Half(double value);
    public static explicit operator float(Half value);

    public int CompareTo(Half other);
    public int CompareTo(object? other);

    public bool Equals(Half other);
    public override bool Equals(object? other);

    public override int GetHashCode();

    public static bool IsFinite(Half value);
    public static bool IsInfinity(Half value);
    public static bool IsNaN(Half value);
    public static bool IsNegative(Half value);
    public static bool IsNegativeInfinity(Half value);
    public static bool IsNormal(Half value);
    public static bool IsPositiveInfinity(Half value);
    public static bool IsSubnormal(Half value);

    public static Half Parse(string s);
    public static Half Parse(string s, NumberStyles style);
    public static Half Parse(string s, IFormatProvider? provider);
    public static Half Parse(string s, NumberStyles style = NumberStyles.AllowThousands | NumberStyles.Float, IFormatProvider? provider = default);
    public static Half Parse(ReadOnlySpan<char> s, NumberStyles style = NumberStyles.AllowThousands | NumberStyles.Float, IFormatProvider? provider = default);

    public override string ToString();
    public string ToString(string? format);
    public string ToString(string? format, IFormatProvider? provider);
    public string ToString(IFormatProvider? provider);

    public bool TryFormat(Span<char> destination, out int charsWritten, ReadOnlySpan<char> format = default, IFormatProvider? provider = default);

    public static bool TryParse(string? s, out Half result);
    public static bool TryParse(string? s, NumberStyles style, IFormatProvider? provider, out Half result);
    public static bool TryParse(ReadOnlySpan<char> s, out Half result);
    public static bool TryParse(ReadOnlySpan<char> s, NumberStyles style, IFormatProvider? provider, out Half result);
}

Polyfill System.MathF in Microsoft.Bcl.MathF

System.MathF was added in .NET Core 2.0 to provide float support that had parity with the double support in System.Math. Given that float is the core computational type used in AI scenarios, many downlevel libraries currently provide their own internal wrappers around System.Math. .NET ships several such shims for its own scenarios and the proposed System.Numerics library would be no exception. As such, it would be beneficial to simply provide this functionality officially and allow such targets to remove their shims. This simplifies their experience and may give additional performance or correctness over the naive approach.

namespace System;

public static class MathF
{
    public const float E = 2.71828183f;
    public const float PI = 3.14159265f;

    public static float Abs(float x);

    public static float Acos(float x);
    public static float Asin(float x);
    public static float Atan(float x);
    public static float Atan2(float y, float x);

    public static float Cos(float x);
    public static float Sin(float x);
    public static float Tan(float x);

    public static float Cosh(float x);
    public static float Sinh(float x);
    public static float Tanh(float x);

    public static float Ceiling(float x);
    public static float Floor(float x);
    public static float Truncate(float x);

    public static float Exp(float x);
    public static float Log(float x);
    public static float Log(float x, float y);
    public static float Log10(float x);

    public static float IEEERemainder(float x, float y);

    public static float Max(float x, float y);
    public static float Min(float x, float y);

    public static float Pow(float x, float y);

    public static float Round(float x);
    public static float Round(float x, int digits);
    public static float Round(float x, int digits, MidpointRounding mode);
    public static float Round(float x, MidpointRounding mode);

    public static int Sign(float x);

    public static float Sqrt(float x);
}

Tagging subscribers to this area: @dotnet/area-system-numerics
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Future of Numerics and AI

.NET provides a broad range of support for various development domains, ranging from the creation of performance-oriented framework code to the rapid development of cloud native services, and beyond. In recent years, especially with the rise of AI and machine learning, there has been a prevalent push towards improving numerical support and allowing developers to more easily write and consume general purpose and reusable algorithms that work with a range of types and scenarios. While .NET's support for scalar algorithms and fixed-sized vectors/matrices is strong and continues to grow, its built-in library support for other concepts, such as tensors and arbitrary length vectors/matrices, is behind the times. Developers writing .NET applications, services, and libraries currently need to seek external dependencies in order to utilize functionality that is considered core or built-in to other ecosystems. In particular, for developers incorporating AI and copilots into their existing .NET applications and services, we strive to ensure that the core numerics support necessary to be successful is available and efficient, and that .NET developers are not forced to seek out non-.NET solutions in order for their .NET projects to be successful.

.NET Framework Compatible APIs

While there are many reasons for developers to migrate to modern .NET and it is the goto target for many new libraries or applications, there remains many large repos that have been around for years where migration can be non-trivial. And while there are many new language and runtime features which cannot ever work on .NET Framework, there still remains a subset of APIs that would be beneficial to provide and which can help those existing codebases bridge the gap until they are able to successfully complete migration.

It is therefore proposed that an out of band System.Numerics.Tensors package be provided which provides a core set of APIs that are compatible with and appropriate for use on .NET Standard 2.0. There are also two types that would be beneficial to polyfill downlevel as part of this process, System.Half and System.MathF, which will significantly improves the usability of the libraries for common scenarios.

Provide the Tensor class

NOTE: The following is an extreme simplification meant to give the general premise behind the APIs being exposed. It is intentionally skimming over several concepts and deeper domain specific terminology that is not critical to cover for the design proposal to be reviewed.

At a very high overview, you have scalars, vectors, and matrices. A scalar is largely just a single value T, a vector is a set of scalars, a matrix is a set of vectors. All of these can be loosely thought of as types of tensors and they have various relationships and can be used to build up and represent higher types. That is, you could consider that T is a scalar, or a vector of length 1, or a 1x1 matrix. Likewise, you can then consider that T[] is a vector or unknown length and that T[,] represents a 2-dimensional matrix and so on. All of these can be defined as types of tensors. -- As a note, this terminology is partially where the name for std::vector<T> in C++ (and similar in other languages) comes from. This is also where the general considerations of "vectorization" when writing SIMD or SIMT arise, although they are typically over fixed-length, rather than arbitrary length.

From the mathematical perspective, many of the operations you can do on scalars also apply to the higher types. There can sometimes be limitations or other considerations that can specialize or restrict these operations, but in principle they tend to exist and work mostly the same way. It is therefore generally desirable to support many such operations more broadly. This is particularly prevalent for "element-wise" operations which could be thought of as effectively doing array.Select((x) => M(x)) and where an explicit API can often provide significant performance improvements over the naive approach.

The core set of APIs described below cover the main arithmetic and conversion operators provided for T as well as element-wise operations for the functionality exposed by System.Math/System.MathF. Some design notes include:

• Where the scalar API Method would return a bool, the vector API has two overloads MethodAny and MethodAll.
• APIS will require that developers utilize Span<T> to access these APIs, overloads taking T[] are not provided
• APIs will support the destination buffer and one of the source buffers being the same.
• APIs taking more than 1 operand will support the other operands being a scalar and it indicating the same value is used for every element.
• In order to make usage of the APIs less "clunky", particularly given these are mathematical APIs where building up complex expressions may be more prevalent, it is proposed a slight deviation from the normal API signature is taken. Rather than simply returning the number of elements written to the destination span, it is proposed that the APIs return a Span<T> instead. This would simply be destination sliced to the appropriate length. This still provides the required information on number of elements written, but gives the additional advantage that the result can be immediately passed into the next user of the algorithm without requiring the user to slice or do length checks themselves.

For targeting modern .NET, there will be a separate future proposal detailing a Tensor<T> type. This then matches a similar split we have for other generic types such as Vector<T> and Vector. The non-generic Tensor class holds extension methods, APIs that are only meant to support a particular set of T, and static APIs. While Tensor<T> will hold operators, instance methods, and core properties. The APIs defined for use on .NET Framework are effectively the "workhorse" APIs that Tensor<T> would then delegate to. They more closely resemble the signatures from the BLAS and LAPACK libraries, which are the industry standard baseline for Linear Algebra operations and allow tensor like functionality to be supported for arbitrary memory while allowing modern .NET to provide a type safe and friendlier way to work with such functionality that can simultaneously take advantage of newer language/runtime features, such as static virtuals in interfaces or generic math.

namespace System.Numerics.Tensors;

public static partial class Tensor
{
    // Element-wise Arithmetic

    public static Span<float> Add(ReadOnlySpan<float> left, ReadOnlySpan<float> right, Span<float> destination);
    public static Span<float> Add(ReadOnlySpan<float> left, T right, Span<float> destination);

    public static Span<float> Subtract(ReadOnlySpan<float> left, ReadOnlySpan<float> right, Span<float> destination);
    public static Span<float> Subtract(ReadOnlySpan<float> left, T right, Span<float> destination);

    public static Span<float> Multiply(ReadOnlySpan<float> left, ReadOnlySpan<float> right, Span<float> destination);
    public static Span<float> Multiply(ReadOnlySpan<float> left, T right, Span<float> destination); // BLAS1: scal

    public static Span<float> Divide(ReadOnlySpan<float> left, ReadOnlySpan<float> right, Span<float> destination);
    public static Span<float> Divide(ReadOnlySpan<float> left, T right, Span<float> destination);

    public static Span<float> Negate(ReadOnlySpan<float> values, Span<float> destination);

    public static Span<float> AddMultiply(ReadOnlySpan<float> left, ReadOnlySpan<float> right, ReadOnlySpan<float> multiplier, Span<float> destination);
    public static Span<float> AddMultiply(ReadOnlySpan<float> left, ReadOnlySpan<float> right, T multiplier, Span<float> destination);
    public static Span<float> AddMultiply(ReadOnlySpan<float> left, T right, ReadOnlySpan<float> multiplier, Span<float> destination);

    public static Span<float> MultiplyAdd(ReadOnlySpan<float> left, ReadOnlySpan<float> right, ReadOnlySpan<float> addend, Span<float> destination);
    public static Span<float> MultiplyAdd(ReadOnlySpan<float> left, ReadOnlySpan<float> right, T addend, Span<float> destination); // BLAS1: axpy
    public static Span<float> MultiplyAdd(ReadOnlySpan<float> left, T right, ReadOnlySpan<float> addend, Span<float> destination);

    public static Span<float> Exp(ReadOnlySpan<float> x, Span<float> destination);
    public static Span<float> Log(ReadOnlySpan<float> x, Span<float> destination);

    public static Span<float> Cosh(ReadOnlySpan<float> x, Span<float> destination);
    public static Span<float> Sinh(ReadOnlySpan<float> x, Span<float> destination);
    public static Span<float> Tanh(ReadOnlySpan<float> x, Span<float> destination);

    // Vector Arithmetic

    // A measure of similarity between two non-zero vectors of an inner product space.
    // It is widely used in natural language processing and information retrieval.
    public static float CosineSimilarity(ReadOnlySpan<float> x, ReadOnlySpan<float> y);

    // A measure of distance between two points in a Euclidean space.
    // It is widely used in mathematics, engineering, and machine learning.
    public static float Distance(ReadOnlySpan<float> x, ReadOnlySpan<float> y);

    // A mathematical operation that takes two vectors and returns a scalar.
    // It is widely used in linear algebra and machine learning.
    public static float Dot(ReadOnlySpan<float> x, ReadOnlySpan<float> y); // BLAS1: dot

    // A mathematical operation that takes a vector and returns a unit vector in the same direction.
    // It is widely used in linear algebra and machine learning.
    public static float Normalize(ReadOnlySpan<float> x); // BLAS1: nrm2

    // A function that takes a collection of real numbers and returns a probability distribution.
    // It is widely used in machine learning and deep learning.
    public static float SoftMax(ReadOnlySpan<float> x);

    // A function that takes a real number and returns a value between 0 and 1.
    // It is widely used in machine learning and deep learning.
    public static Span<float> Sigmoid(ReadOnlySpan<float> x, Span<float> destination);

    public static float Max(ReadOnlySpan<float> value);
    public static float Min(ReadOnlySpan<float> value);

    public static int IndexOfMaxMagnitude(ReadOnlySpan<float> value); // BLAS1: iamax
    public static int IndexOfMinMagnitude(ReadOnlySpan<float> value);

    public static float Sum(ReadOnlySpan<float> value);
    public static float SumOfSquares(ReadOnlySpan<float> value);
    public static float SumOfMagnitudes(ReadOnlySpan<float> value); // BLAS1: asum

    public static float Product(ReadOnlySpan<float> value);
    public static float ProductOfSums(ReadOnlySpan<float> left, ReadOnlySpan<float> right);
    public static float ProductOfDifferences(ReadOnlySpan<float> left, ReadOnlySpan<float> right);

    // Vector Conversions

    public static Span<Half> ConvertToHalf(ReadOnlySpan<float> value, Span<Half> destination);
    public static Span<float> ConvertToSingle(ReadOnlySpan<Half> value, Span<float> destination);
}

Polyfill System.Half in Microsoft.Bcl.Half

System.Half is a core interchange type for AI scenarios, often being used to minify the storage impact for the tens of thousands to millions of data points that need to be stored. It is not, however, as frequently used for computation.

We initially exposed this type in .NET 5 purely as an interchange type and it is therefore lacking the arithmetic operators. These operators were later added in .NET 6/7 as a part of the generic math initiative. For .NET Framework, the interchange surface area should be sufficient and will follow the general guidance required for polyfills that they meet the initial shape we shipped, even if the version it shipped on is no longer in support (that is, the .NET Standard 2.0 surface area needs to remain compatible with .NET 5, even though .NET 5 is out of support).

namespace System;

public readonly struct Half : IComparable, IComparable<Half>, IEquatable<Half>, IFormattable
{
    public static Half Epsilon { get; }
    public static Half MaxValue { get; }
    public static Half MinValue { get; }
    public static Half NaN { get; }
    public static Half NegativeInfinity { get; }
    public static Half PositiveInfinity { get; }

    public static bool operator ==(Half left, Half right);
    public static bool operator !=(Half left, Half right);

    public static bool operator >(Half left, Half right);
    public static bool operator >=(Half left, Half right);

    public static bool operator <(Half left, Half right);
    public static bool operator <=(Half left, Half right);

    public static explicit operator double(Half value);
    public static explicit operator Half(float value);
    public static explicit operator Half(double value);
    public static explicit operator float(Half value);

    public int CompareTo(Half other);
    public int CompareTo(object? other);

    public bool Equals(Half other);
    public override bool Equals(object? other);

    public override int GetHashCode();

    public static bool IsFinite(Half value);
    public static bool IsInfinity(Half value);
    public static bool IsNaN(Half value);
    public static bool IsNegative(Half value);
    public static bool IsNegativeInfinity(Half value);
    public static bool IsNormal(Half value);
    public static bool IsPositiveInfinity(Half value);
    public static bool IsSubnormal(Half value);

    public static Half Parse(string s);
    public static Half Parse(string s, NumberStyles style);
    public static Half Parse(string s, IFormatProvider? provider);
    public static Half Parse(string s, NumberStyles style = NumberStyles.AllowThousands | NumberStyles.Float, IFormatProvider? provider = default);
    public static Half Parse(ReadOnlySpan<char> s, NumberStyles style = NumberStyles.AllowThousands | NumberStyles.Float, IFormatProvider? provider = default);

    public override string ToString();
    public string ToString(string? format);
    public string ToString(string? format, IFormatProvider? provider);
    public string ToString(IFormatProvider? provider);

    public bool TryFormat(Span<char> destination, out int charsWritten, ReadOnlySpan<char> format = default, IFormatProvider? provider = default);

    public static bool TryParse(string? s, out Half result);
    public static bool TryParse(string? s, NumberStyles style, IFormatProvider? provider, out Half result);
    public static bool TryParse(ReadOnlySpan<char> s, out Half result);
    public static bool TryParse(ReadOnlySpan<char> s, NumberStyles style, IFormatProvider? provider, out Half result);
}

Polyfill System.MathF in Microsoft.Bcl.MathF

System.MathF was added in .NET Core 2.0 to provide float support that had parity with the double support in System.Math. Given that float is the core computational type used in AI scenarios, many downlevel libraries currently provide their own internal wrappers around System.Math. .NET ships several such shims for its own scenarios and the proposed System.Numerics library would be no exception. As such, it would be beneficial to simply provide this functionality officially and allow such targets to remove their shims. This simplifies their experience and may give additional performance or correctness over the naive approach.

namespace System;

public static class MathF
{
    public const float E = 2.71828183f;
    public const float PI = 3.14159265f;

    public static float Abs(float x);

    public static float Acos(float x);
    public static float Asin(float x);
    public static float Atan(float x);
    public static float Atan2(float y, float x);

    public static float Cos(float x);
    public static float Sin(float x);
    public static float Tan(float x);

    public static float Cosh(float x);
    public static float Sinh(float x);
    public static float Tanh(float x);

    public static float Ceiling(float x);
    public static float Floor(float x);
    public static float Truncate(float x);

    public static float Exp(float x);
    public static float Log(float x);
    public static float Log(float x, float y);
    public static float Log10(float x);

    public static float IEEERemainder(float x, float y);

    public static float Max(float x, float y);
    public static float Min(float x, float y);

    public static float Pow(float x, float y);

    public static float Round(float x);
    public static float Round(float x, int digits);
    public static float Round(float x, int digits, MidpointRounding mode);
    public static float Round(float x, MidpointRounding mode);

    public static int Sign(float x);

    public static float Sqrt(float x);
}

Author:	tannergooding
Assignees:	-
Labels:	area-System.Numerics, api-ready-for-review
Milestone:	-

[jkotas]

It is therefore proposed that an out of band System.Numerics.Tensors package be provided

What is going to happen with the existing content of System.Numerics.Tensors package that never shipped a stable version?

[tannergooding]

What happens with the existing content of System.Numerics.Tensors package that never shipped a stable version?

They will be removed and exist purely in source control history for the near term.

The general design of the types in the existing preview package isn't incorrect. However, they was done as a prototype and done several releases ago at this point, so it doesn't take into account newer features such as generic math, vectorization support, etc. We've also since determined that we do in fact need to provide basic operation support as we've hadmultiple requests from both first and third party customers.

What we're doing here is providing the foundations on which the package can be stabilized and shipped properly. With the initial stabilization being done for .NET Standard to answer some core needs. The Tensor<T> type (and corresponding support for DenseTensor/SparseTensor) will then be redone (building off what's already been done) taking into account recent .NET features and will be only available for modern .NET (targeting .NET 9/10). So they will come back in a future version of the package.

[jkoritzinsky]

Can we rename the Distance methods to EuclideanDistance to make sure it's clear which definition of distance they're using (as there's many different kinds of "distance" in Euclidean spaces)?

[gfoidl]

Sigmoid should be more specific, i.e. what sigmoid-function it's exactly -- e.g. SigmoidLogistic?

For the tensor operations should there be double operations as well?
(For my usecases double would be preferable)

[gfoidl]

Re: EuclideanDistance perf-wise it could be the squared distance, as the (square) root is monotonic, so for comparisons (i.e. what's the nearest) one doesn't need to calculate the square root.
If so, maybe name it EuclideanDistance2?

[tannergooding]

Can we rename the Distance methods to EuclideanDistance to make sure it's clear which definition of distance they're using (as there's many different kinds of "distance" in Euclidean spaces)?

We could, but we already define other APIs named Distance with them being the standard "pythagorean distance" (or straight line distance) that is most typical. I imagine other distances are then the ones that would need/get additional clarification as to their kind.

This is also common in other libraries/spaces. If we were to prefix it as EuclideanDistance, it would set precedent for other APIs such as Normalize to also be prefixed or named differently.

Re: EuclideanDistance perf-wise it could be the squared distance, as the (square) root is monotonic, so for comparisons (i.e. what's the nearest) one doesn't need to calculate the square root.
If so, maybe name it EuclideanDistance2?

It would have to be named *Squared to match the existing convention we use elsewhere. 2 itself is ambiguous. That being said, the cost of the square root tends to be trivial, particularly for large inputs, and its likely not worth it except for very small inputs.

We could also expose EuclideanDistanceSquared or DistanceSquared, but that may "open the floodgates" to doing this in many places.

Sigmoid should be more specific, i.e. what sigmoid-function it's exactly -- e.g. SigmoidLogistic?

Same here as with "Euclidean Distance". Many libraries simply expose the core SigmoidLogistic function as simply Sigmoid and only differentiate other variants as necessary.

There's a balance between clarity and matching existing industry standards we want to reach.

[tannergooding]

For the tensor operations should there be double operations as well?
(For my usecases double would be preferable)

Double isn't on the table, currently at least, for the .NET Standard support. It is very much on the table for the modern .NET support which will utilize generics and likely generic math.

[gfoidl]

There's a balance between clarity and matching existing industry standards we want to reach.

I agree for the distance part. Keep Distance as the dominant one is the euclidean distance. Other distance measures could be given as separate overload if there's demand -- or maybe better abstract it with IDistance and generics, which defaults to the euclidean one, others can be given easily. But that won't work for .NET Standard.

For the sigmoid part I disagree. There are too many of them available and used, each with somewhat different characteristics.
Although the name "sigmoid" is used by many libraries, one has to look into the docs to see what function exactly is implemented -- if it's documented.
So why not make it obvious by adding the concrete function type to the name?
Also I don't see a problem be being incosistent in the naming of Distance (w/o specifying the conrete euclidean distance) and SigmoidLogistic (w/ specifying the concrete type), as "distance" is in general refered to the euclidean one, whilst for sigmoid it's a group of functions with characteristic S-shape.

This is also common in other libraries/spaces.

Well...if other libraries have it that way it doesn't mean it's correct and that .NET has to follow along...if could be done better 😉 (I hope you get how I mean this)

[tannergooding]

Well...if other libraries have it that way it doesn't mean it's correct and that .NET has to follow along...if could be done better 😉 (I hope you get how I mean this)

The consideration is that while it is technically an entire class, there is effectively a domain standard that simply "sigmoid" by itself is the logistic function.

This is true in Intel MKL, Apple Accelerate, Wolfram Alpha and many other libraries oriented towards Linear Algebra (BLAS/LAPACK), as well as Machine Learning and math reference sources.

It is the de-facto response when you search "sigmoid function programming", the reference image used in almost any explanation of what a sigmoid is, and so on. Being different in this case is likely to hurt things more than it would help, particularly for the domain of users that are most likely to use these types; particularly if they have ever done similar in another language or domain. Simply because sigmoid(x) is almost definitively treated as y=1/(1+e^(-x))

Notably it is also what is done on many scientific or graphic calculators which support it.