Can this initcontainer image be configurable?

[pacoxu]

llmaz/pkg/defaults.go

Line 20 in 14dde96

LOADER_IMAGE = "inftyai/model-loader:v0.0.10"

My env cannot pull docker hub image properly. I need to customize this image.

Can we add a ENV: IMAGE_PREFIX or something to make this configurable?

[googs1025]

Maybe we can configure it at the controller flags (not sure) or env?

[kerthcet]

Make sense, we can make it configurable. Let's load it from the env for simplify. Once MODEL_LOADER_IMAGE exists, use the value instead, we can set the env via playground.backendConfig.env, wdyt?

[lazybearlee]

Hi @pacoxu

Reading through the discussion on making the model-loader image configurable, the suggestion to use an environment variable (MODEL_LOADER_IMAGE) is definitely a valid way to move away from the hardcoded value.

I wanted to suggest an alternative approach, drawing inspiration from how similar configurations are often handled in other Kubernetes-native ML serving systems like KServe. We could consider using a ConfigMap to store the model-loader image name (and potentially other default image configurations in the future).

Here's how it could work:

1. Create a ConfigMap: Define a ConfigMap, perhaps named llmaz-config or llmaz-defaults-config, within the llmaz installation namespace. This ConfigMap would hold key-value pairs, for example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: llmaz-defaults-config
     namespace: llmaz-system # Or wherever llmaz controller runs
   data:
     modelLoaderImage: "inftyai/model-loader:v0.0.10"
     # Potentially other configurable images/defaults here later

2. Read from ConfigMap: The llmaz controller (or potentially a mutating admission webhook if llmaz uses one for pod creation/modification, similar to KServe's injector) would be responsible for reading the modelLoaderImage key from this ConfigMap.

   • When the controller reconciles a resource that requires the model loader, instead of using the hardcoded default or an environment variable baked into its own pod spec, it fetches the value from the specified key in the ConfigMap.
   • A default value within the code could still be used as a fallback if the ConfigMap or the specific key is not found.

Advantages of this approach:

• Dynamic Updates: Cluster administrators can update the model-loader image (or other future configurable defaults) by simply editing the ConfigMap, without needing to restart the llmaz controller pods (assuming the controller reads the ConfigMap dynamically or watches it). This is generally more flexible than environment variables, which often require a pod restart to take effect.
• Centralized Configuration: Provides a standard Kubernetes way to manage llmaz-specific configurations centrally, rather than scattering them as environment variables on the controller deployment.
• Separation of Concerns: Separates configuration data from the controller's runtime environment variables.
• Extensibility: Easy to add more configurable default images or settings to the same ConfigMap in the future without changing the controller's deployment spec.

This pattern aligns well with common Kubernetes operator practices and offers significant flexibility for users needing to customize images due to registry restrictions (like the original issue raised) or wanting to use custom-built versions.

The relevant and instructive Kserve code can be found at kserve

[kerthcet]

Thanks @lazybearlee this is definitely a long-term solution, what I proposed above is just a quick temporary fix, would you like to push this forward?

[googs1025]

I understand that we can first provide a quick option, such as this guide.

Make sense, we can make it configurable. Let's load it from the env for simplify. Once MODEL_LOADER_IMAGE exists, use the value instead, we can set the env via playground.backendConfig.env, wdyt?

If the number of config param increases in the future, we can consider cm or other more flexible ways. 😄

[lazybearlee]

@kerthcet Thanks a lot for the positive feedback and for considering the ConfigMap approach!

I completely agree that using an environment variable as proposed is a great pragmatic step for a quick fix right now, especially to unblock the immediate need. It makes sense. Regarding pushing the ConfigMap solution forward – I'm definitely interested and would be happy to contribute to it!

However, as I'm currently in the process of learning more about llmaz and familiarizing myself with the overall codebase, my bandwidth is a bit limited at this very moment to take on implementing this feature immediately. Once I'm more up to speed, I'd be very keen to revisit this ConfigMap enhancement and potentially help drive its implementation, perhaps as a follow-up improvement after the initial environment variable fix is in place.

[kerthcet]

With #431, we have a global configmap now, which I believe we can simply achieve this.

[rudeigerc]

/assign