Automatically created topics have no valid schema info

[BewareMyPower]

I have encountered schema problems recently. eg, creating a consumer with Schema.STRING:

        try (PulsarClient client = PulsarClient.builder() //
                .serviceUrl("pulsar://localhost:6650") //
                .build()) {
            try (Consumer<String> consumer = client.newConsumer(Schema.STRING) //
                    .topic("Foo") //
                    .subscriptionName("test-sub") //
                    .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest) //
                    .subscribe()) {
                log.info("Subscribe successfully");
            }
        } catch (PulsarClientException e) {
            log.error("{}", e.getMessage(), e);
        }

The log is:

org.apache.pulsar.client.api.PulsarClientException$IncompatibleSchemaException: Topic does not have schema to check

Finally I found some topics may not have associated schema, see SchemaRegistryServiceImpl in broker.service.schema package:

    public CompletableFuture<Void> checkConsumerCompatibility(String schemaId, SchemaData schemaData,
                                                              SchemaCompatibilityStrategy strategy) {
        return getSchema(schemaId).thenCompose(existingSchema -> {
            if (existingSchema != null && !existingSchema.schema.isDeleted()) {
                // ...
            } else {  // existingSchema is null or it's deleted
                return FutureUtil.failedFuture(new IncompatibleSchemaException("Topic does not have schema to check"));
            }
        });
    }

I thought it's because those topics have been produced by C++ client without schema. But it seems that I'm wrong. I created a new topic and used C++ client to produce some messages, but a consumer with string schema subscribed successfully. I've also trying producing some messages with Java producer without schema, but the result were the same.

In my private cluster for test, there're still 2 topics that couldn't be subscribed with string schema. But I didn't know how did I make it happen.

[codelipenghui]

@BewareMyPower Which version of the broker are you using?

[BewareMyPower]

@codelipenghui It's currently 2.6.0, but there's an update from 2.5.1 to 2.6.0 a few days ago.

[sijie]

In my private cluster for test, there're still 2 topics that couldn't be subscribed with string schema. But I didn't know how did I make it happen.

Can you run pulsar-admin schemas get to get schemas of these 2 topics? If these 2 topics don't have any schemas, it is expected to fail with Incompatible schema if you try to subscribe to them using Schema.STRING.

[BewareMyPower]

@sijie The results are both HTTP 404 Not Found. The broker's log is also 404

10:39:40.284 [ForkJoinPool.commonPool-worker-14] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [04/Aug/2020:10:39:40 +0800] "GET /admin/v2/schemas/public/default/ParBar/schema HTTP/1.1" 404 0 "-" "Pulsar-Java-v2.6.0" 4

[BewareMyPower]

@codelipenghui @sijie I reproduced the issue accidentally. If the broker's allowAutoTopicCreationType is partitioned, then the automatically created topic has no schema.

It's easy to reproduce:

1. Run a consumer to subscribe a topic that doesn't exist now, e.g.

           try (Consumer<byte[]> consumer = client.newConsumer() //
                   .topic("PulsarBar") // PulsarBar doesn't exist
                   .subscriptionName("my-sub") //
                   .subscribe()) {

2. Check the topic's schema

   $ ./bin/pulsar-admin schemas get "persistent://public/default/PulsarBar"
   HTTP 404 Not Found
   
   Reason: HTTP 404 Not Found
   $ ./bin/pulsar-admin schemas get "persistent://public/default/PulsarBar-partition-0"
   HTTP 404 Not Found
   
   Reason: HTTP 404 Not Found

The broker logs:

11:36:15.409 [ForkJoinPool.commonPool-worker-25] INFO  org.eclipse.jetty.server.RequestLog - 10.208.65.74 - - [09/Sep/2020:11:36:15 +0800] "GET /admin/v2/schemas/public/default/PulsarBar/schema?authoritative=true HTTP/1.1" 404 0 "-" "Pulsar-Java-v2.6.0" 4

[sijie]

@BewareMyPower If you have identified the problem, are you interested in contributing a fix?

[BewareMyPower]

@sijie Yes, I'll take some time to look into the problem.

[BewareMyPower]

The description I've said before is not accurency. If I send a PUT request to create a topic, querying the topic's schema will also returns 404. But a consumer with string schema could still consume it. After that, the topic's schema became string schema.

However, if a topic was created automatically by a consumer without schema, then a consumer with string schema would fail with Topic does not have schema to check.

I'm a little confused with

If these 2 topics don't have any schemas, it is expected to fail with Incompatible schema if you try to subscribe to them using Schema.STRING.

I've read some source code, if the broker only handled admin's create topic request, it would just call BrokerService.getTopic and wait until the returned future completed. But if the broker handled client's subscribe command while the topic didn't exist, the broker would do some more, which might lead to the difference behavior.

It looks like a topic without schema has two different states. I'm not familiar with schema, could you give me some help? @sijie

[sijie]

Discussed with @BewareMyPower, we might need to improve the documentation.