Feature request: Multiple service metadata decorators in a single client

[tjprescott]

OpenAI's scenario is that they need a single client that can make calls either to AzureOpenAI or "regular" OpenAI. These two services share largely the same API structure, models etc (by design) but they different by their hosts and the actual individual API routes which exceeds the capacity of the @server parameter. Simply packaging two clients into a single SDK has already been considered and rejected.

The team is requesting some kind of mechanism to allow routes to "switch" based on the host. This is currently accomplished through manual code.

[timotheeguerin]

Does @server with a parameterized host not fit the need?
https://cadlplayground.z22.web.core.windows.net/?c=aW1wb3J0ICJAdHlwZXNwZWMvaHR0cCI7Cgp1c2luZyBUeXBlU3BlYy5IdHRwOwpAc2VydmljZcYJZXIoCiAgIntob3N0fSIsxAxPcGVuQVBJIMYfxRR7CiAgICDEJDogdXJsxBN9CikKbmFtZXNwYWNlIERlbW9Txlg7CgpvcCB0ZXN0KCk6IHN0cmluZzsK

[tjprescott]

No, it doesn't. It's not just the host that differs. While the APIs are the same, the Azure and non-Azure routes are different.

[jpalvarezl]

Hi everyone, I am here just to provide concrete examples, hoping this helps to specify the context better of this request.

We are currently trying to support both Azure and non-Azure OpenAI backends with our SDKs. We currently are working with several teams to generate SDKs, for the following languages (so far):

• .NET
• JavaScript
• Java

For the case of .NET and JavaScript the support, under a single client SDK, for both Azure and non-Azure OpenAI services is already there. In the case of .NET, we had to override the GetUri method and alter the request URL to point to the right server. You can see the code here. For JavaScript, the split in behaviour happens here.

Java has just started development, for non-Azure OpenAI support I have drafted a PR that somewhat showcases some of the challenges of this "2 servers/1 client" setup.

Mainly, Azure and non-Azure OpenAI:

• Don't share version names (non-Azure OpenAI only has v1, that I know of)
• Versions are provided totally different. non-AOAI the version name is part of the path, whereas AOAI it's a query parameter.
• Partly because of the last point, route paths are totally different.

One proposal surfaced in a meeting today by @tjprescott , that I personally liked, was the possibility of simply splitting the 2 services into 2 clients and just add enough custom code to have a single client wrapper over the 2. I think this would address all the issues while seemingly keeping the manual maintenance cost relatively low. Although, I could be missing something. I share this here too so we don't lose visibility of all the alternatives, and so that we can also take this into account for the assessment of the feasibility of this feature.

[timotheeguerin]

If I understand correctly the url are something of the sort:

• "azure-host.com/{path}?api-version${api-version}",
• "openapi-host.com/v1/{path}"

Could imagine this where we provide the query param as part of the server path?

import "@typespec/http";

using TypeSpec.Http;
@service
@server(
  "azure.com/?api-version${api-version}",
  "OpenAPI server",
  {
    apiVersion: "2023-01-01" | "2023-02-01",
  }
)
@server(
  "openapi.com",
  "OpenAPI server",
  {
    apiVersion: "v1",
  }
)
namespace DemoService;

op test(): string;

Playground

[jpalvarezl]

I realise I could have shared the tsp definition where we generate the code from. Here is the service definition.

There is also @useAuth to take into consideration. Technically, AOAI supports AAD and Token, but non-AzureOAI uses only Token auth.

An example of how routes differ:

• For Azure OpenAI

{
      "RequestUri": "https://openai.azure.com/openai/deployments/text-davinci-002/completions?api-version=2023-03-15-preview"
  ...

• For non-Azure OpenAI:

{
      "RequestUri": "https://api.openai.com/v1/completions",
...

These would return the same object type.

[timotheeguerin]

I see, so there is also the deployments that is present in the Azure api but no OpenAI. In the spec this is defined as an operation level parameter, what do we expect the generated client to look like? Should it have that deployment?

function getCompletions(deployment: string) {

}

// or deployment is provided at the client level?

function getCompletions() {

}

@useAuth being different does make this a little harder too.

[jpalvarezl]

This is one of the bigger pain points, or rather confusion points. deploymentId while provided as part of the path for Azure OpenAI, in the non-Azure OpenAI case, it is provided under CompletionsOptions.modelName field in the request object (this is a POST operation). You can see the forking custom behaviour in this case in .NET over here

Note: the field is renamed to nonAzureModel

[jpalvarezl]

I brought up the @userAuth it's just to illustrate some of the challenges, but it is something that certainly we don't need to address in this feature request, as far as I understand. The more pressing issue is the split between route definition for the services.