Namespace seeding does not work

[ericlacher]

I tried this:

I was trying to seed a namespace by file when launching the API

metadata:
  name: mynamespace

This namespace never would show up in the ui.

This happened:

Namespace never appeared in UI, but first attempts succeeds (2nd fails duew to duplicate name in resource repository, I believe due to absence of a name).

I expected this:

namespace to appear in UI

Is there a workaround?

I altered these lines to this:

var ns = serializer......
var ns_resource = new Namespace(){
  Metadata = new(){
    Name = ns.Metadata.Name
  }
};

Which does work for me. I am not a .Net/C# developer, so there might be better ways to do this (e.g. ProvisionWorkflowsAsync has more sophisiticated checks for duplicates etc.), but kindly asking for someone to fix it.

Anything else?

No response

Platform(s)

MacOS

Community Notes

• Please vote by adding a 👍 reaction to the issue to help us prioritize.
• If you are interested to work on this issue, please leave a comment.name: Bug Report 🐞

[cdavernas]

@ericlacher Thanks for reporting, I'll have a look in next week after returning from PTO. It looks like a bug, not sure what happens.

In the meanwhile, could you be so kind to provide the whole content of your namespace YAML file?

If what you supplied above is the entire content of said file, then the issue is likely related to missing kind and apiVersion fields.

[ericlacher]

@cdavernas the snippet I sent actually is the full namespace yaml I used.
However, I also tried it with apiVersion etc., without success.
It seems the NamespaceDefinition class is not suitable for deserialization or adding to a repository. Changing DatabaseProvisioner to use Namespace class instead did it for me. (unrelated: the file DatabaseProvisioner .cs has a whitespace before the dot).

Attempts

I tried using difference combinations of upper/lower cases, as sometimes the kind (Namespace) and sometimes the name was missing.

lower case metadata only

input:

metadata:
  name: foo

seeding 1st attempt

api-1  | info: Synapse.Api.Application.Services.WorkflowDatabaseInitializer[0]
api-1  |       Successfully imported namespace 'foo' from file '/app/seed/namespaces/foo/foo.yaml'
api-1  | info: Synapse.Api.Application.Services.WorkflowDatabaseInitializer[0]
api-1  |       Completed importing 1 namespaces in 7.221 milliseconds

seeding again

An error occurred while reading a namespace from file '/app/seed/namespaces/foo/foo.yaml': Neuroglia.ProblemDetailsException: [400 - Conflict] The specified resource 'neuroglia.io/v1/resource-definitions/foo' already exists
api-1  |          at Neuroglia.Data.Infrastructure.ResourceOriented.Services.RedisDatabase.CreateResourceAsync(IResource resource, String group, String version, String plural, String namespace, Boolean dryRun, CancellationToken cancellationToken) in /home/runner/work/framework/framework/src/Neuroglia.Data.Infrastructure.ResourceOriented.Redis/Services/RedisDatabase.cs:line 128

-> weird resource string (namespace missing): neuroglia.io/v1/resource-definitions/foo
-> Namespace not showing up in UI.

Upper Case

input:

ApiVersion: neuroglia.io/v1
Kind: Namespace
Metadata:
  Name: foo

Seeding

api-1  | info: Synapse.Api.Application.Services.WorkflowDatabaseInitializer[0]
api-1  |       Starting importing namespaces from directory '/app/seed/namespaces'...
api-1  | fail: Synapse.Api.Application.Services.WorkflowDatabaseInitializer[0]
api-1  |       An error occurred while reading a namespace from file '/app/seed/namespaces/foo/foo.yaml': Neuroglia.ProblemDetailsException: [400 - Conflict] The specified resource 'neuroglia.io/v1/resource-definitions/namespaces.neuroglia.io' already exists

-> even worse resource identifier I think (name missing now): neuroglia.io/v1/resource-definitions/namespaces.neuroglia.io
-> Namespace not showing up in UI.

Mixed cases

input:

ApiVersion: neuroglia.io/v1
Kind: Namespace
metadata:
  name: foo

seeding:

api-1  | info: Synapse.Api.Application.Services.WorkflowDatabaseInitializer[0]
api-1  |       Successfully imported namespace 'foo' from file '/app/seed/namespaces/foo/foo.yaml'
api-1  | info: Synapse.Api.Application.Services.WorkflowDatabaseInitializer[0]
api-1  |       Completed importing 1 namespaces in 7.1403 milliseconds

seeding again

An error occurred while reading a namespace from file '/app/seed/namespaces/foo/foo.yaml': Neuroglia.ProblemDetailsException: [400 - Conflict] The specified resource 'neuroglia.io/v1/resource-definitions/foo' already exists
api-1  |          at Neuroglia.Data.Infrastructure.ResourceOriented.Services.RedisDatabase.CreateResourceAsync(IResource resource, String group, String version, String plural, String namespace, Boolean dryRun, CancellationToken cancellationToken) in /home/runner/work/framework/framework/src/Neuroglia.Data.Infrastructure.ResourceOriented.Redis/Services/RedisDatabase.cs:line 128

-> weird resource descriptor (namespace missing): neuroglia.io/v1/resource-definitions/foo
-> Namespace not showing up in UI.

Outcome

So, no matter what I did, I was not able to seed the namespace, despite log messagres indicating it was imported (most likely due to wrong assembling of resource identifer string).

Working Examples for Seeding:

In general I'd say the format of the to-be-seeded files could deserve a mention in the docs (or did I miss this?).
This is what I've tested and what seems to work:

Functions: /functions/**/*.yaml (multiple versions per file)

apiVersion: synapse.io/v1
kind: CustomFunction
metadata:
  name: foo
spec:
  versions:
    - 1.0.0:
      run:
        ...

Workflows: /workflows/**/*.yaml (single version per file)

document:
  dsl: "1.0.0"
  namespace: default
  name: bar
  version: "0.1.0"
do:
  ...

Namespace: /namespaces/**/*.yaml (for me, this only works when patching DatabaseProvisioner .cs as described)

metadata:
  name: baz

[ericlacher]

As another sidenote, I am trying to use namespaces to route workflow execution to specific operators. As we're going to deploy to k8s, I currently expect different loads required to be executed on different nodes (e.g. GPU loads vs CPU intensive loads vs low-profile loads). In the docs, this is not described in detail. While I'd go with namespaces to separate domains either way, I'd be interested to know whether there's better ways to achieve routing workloads to nodes (e.g. labels/tags?).

[cdavernas]

It seems the NamespaceDefinition class is not suitable for deserialization or adding to a repository.

@ericlacher You are right, it's a typo, should be Namespace and not NamespaceDefinition.

As another sidenote, I am trying to use namespaces to route workflow execution to specific operators.

You can use labels to do so. I'll make sure to update the docs when back. In the meanwhile, you can configure that in the operator's spec.selector

[ericlacher]

Thanks @cdavernas, if you find some time, I'd appreciate more detailed insights into routing workflow execution.
From your answer above, I assume spec.selector will be used as a normal k8s nodeSelector, so assuming I have a node with label foo:bar, I could set

...
spec:
  selector:
    foo: "bar"
  ...

That would solve binding certain operators to specific nodes.

However, how can I make sure a certain workflow is always getting executed by specific operators? Is there a similar concept? E.g. I found nothing on the workflow definition, but there's tags on the document.

As mentioned, I think namespaces are doing this for me, but perhaps there's another way to achieve this additionally.

[ericlacher]

the routing discussion was taken up separately in a discussion.