[Doc][improve] Auto-detect broken links and integrate into CI process

[labuladong]

Search before asking

• I searched in the issues and found nothing similar.

What issue do you find in Pulsar docs?

After #17495 and #17599, we can see many broken links in the Pulsar document. And more links may break during further evolution.

What is your suggestion?

In #17599 I use a script but it's not reliable enough so I have to check the links list manually. I suggest developing a reliable script to auto-detect the incorrect links in the Pulsar document. Maybe we can integrate this script into the CI process which is related to document change.

Before development, I'd like to enumerate all kinds of broken links.

1. wrong markdown file reference

For example the link of this page:

The markdown content is [configuration](reference-configuration.md), but the reference-configuration.md file not exists.

2. 404 URL path

For example the link of this page:

The markdown content is [type](/api/client/index.html?org/apache/pulsar/client/api/CompressionType.html), but the Pulsar site doesn't have this path.

3. confusing URL path

For example the link of this page:

The markdown content is [Pulsar Functions CLI](/tools/pulsar-admin/), but this refers to a confusing page:

4. invalid title anchor

We can use # to refer to a specific block of HTML this way: [dataDir](reference-configuration.md#zookeeper-dataDir). So if our script can detect the anchor will be better.

Our script should be able to detect these broken links and print warning messages to users.

cc @tisonkun @Anonymitaet @momo-jun @michaeljmarshall

Any reference?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[tisonkun]

Since we build some pages on the fly instead of all content static, we may need an external sensor/crawler for the online version, or static check after the site was built.

Related issue: #18014

[Anonymitaet]

For external tools, normally we use https://www.drlinkcheck.com/ to check URLs.

[labuladong]

#18014 Looks great, but it seems like a big change. If we build the site dynamically, I think detecting the generated HTML files may be better than the external crawler.

Before #18014 finishes, I think 1. wrong markdown file reference and 4. invalid title anchors are still worth detecting.

[tisonkun]

For external tools, normally we use https://www.drlinkcheck.com/ to check URLs.

It helps detect this issue apache/pulsar-site#252, PTAL.

[labuladong]

The detect script see here. PR #18109 fix almost all invalid title anchors. But there are still many invalid markdown file references about reference-configuration.md:

administration-pulsar-shell.md has 1 errors:

[pulsar-client](reference-cli-tools#pulsar-client.md) -> reference-cli-tools#pulsar-client.md not exist

administration-geo.md has 2 errors:

[per-tenant basis](#concepts-multi-tenancy) -> administration-geo.md doesn't have anchor: #concepts-multi-tenancy
[broker configuration](reference-configuration.md#broker) -> reference-configuration.md not exist

security-athenz.md has 1 errors:

[Broker Configuration](reference-configuration.md#broker) -> reference-configuration.md not exist

client-libraries-websocket.md has 6 errors:

[`webSocketServiceEnabled`](reference-configuration.md#broker-webSocketServiceEnabled) -> reference-configuration.md not exist
[`conf/broker.conf`](reference-configuration.md#broker) -> reference-configuration.md not exist
[`conf/websocket.conf`](reference-configuration.md#websocket) -> reference-configuration.md not exist
[`configurationMetadataStoreUrl`](reference-configuration.md#websocket) -> reference-configuration.md not exist
[`webServicePort`](reference-configuration.md#websocket-webServicePort) -> reference-configuration.md not exist
[`clusterName`](reference-configuration.md#websocket-clusterName) -> reference-configuration.md not exist

security-authorization.md has 4 errors:

[`conf/broker.conf`](reference-configuration.md#broker) -> reference-configuration.md not exist
[Broker Configuration](reference-configuration.md#broker) -> reference-configuration.md not exist
[`conf/broker.conf`](reference-configuration.md#broker) -> reference-configuration.md not exist
[`conf/broker.conf`](reference-configuration.md#broker) -> reference-configuration.md not exist

cookbooks-non-persistent.md has 3 errors:

[`enableNonPersistentTopics`](reference-configuration.md#broker-enableNonPersistentTopics) -> reference-configuration.md not exist
[`standalone.conf`](reference-configuration.md#standalone) -> reference-configuration.md not exist
[`enablePersistentTopics`](reference-configuration.md#broker-enablePersistentTopics) -> reference-configuration.md not exist

security-tls-authentication.md has 1 errors:

[the configuration to enable TLS encryption](security-tls-transport.md#configure-tls-encryption-in-cli-tools) -> security-tls-transport.md doesn't have anchor: #configure-tls-encryption-in-cli-tools

administration-zk-bk.md has 7 errors:

[`conf/zookeeper.conf`](reference-configuration.md#zookeeper) -> reference-configuration.md not exist
[`dataDir`](reference-configuration.md#zookeeper-dataDir) -> reference-configuration.md not exist
[`conf/global_zookeeper.conf`](reference-configuration.md#configuration-store) -> reference-configuration.md not exist
[parameters](reference-configuration.md#zookeeper) -> reference-configuration.md not exist
[`metadataStoreBatchingEnabled`](reference-configuration.md#broker) -> reference-configuration.md not exist
[`conf/bookkeeper.conf`](reference-configuration.md#bookkeeper) -> reference-configuration.md not exist
[`zkServers`](reference-configuration.md#bookkeeper-zkServers) -> reference-configuration.md not exist

client-libraries-node.md has 1 errors:

[these instructions](client-libraries-cpp.md#compilation) -> client-libraries-cpp.md doesn't have anchor: #compilation

admin-api-tenants.md has 1 errors:

[configuration](reference-configuration.md) -> reference-configuration.md not exist

developing-binary-protocol.md has 5 errors:

[`BaseCommand`](#pulsar.proto.BaseCommand) -> developing-binary-protocol.md doesn't have anchor: #pulsar.proto.BaseCommand
[`Type`](#pulsar.proto.Type) -> developing-binary-protocol.md doesn't have anchor: #pulsar.proto.Type
[`brokerEntryMetadataInterceptors`](reference-configuration.md#broker) -> reference-configuration.md not exist
[`brokerEntryMetadataInterceptors`](reference-configuration.md#broker) -> reference-configuration.md not exist
[`exposingBrokerEntryMetadataToClientEnabled`](reference-configuration-broker.md#exposingbrokerentrymetadatatoclientenabled) -> reference-configuration-broker.md not exist

concepts-multiple-advertised-listeners.md has 1 errors:

[broker configuration file](reference-configuration.md#broker) -> reference-configuration.md not exist

admin-api-clusters.md has 1 errors:

[configuration](reference-configuration.md) -> reference-configuration.md not exist

concepts-architecture-overview.md has 2 errors:

[non-persistent](concepts-messaging.md#non-persistent-topics.md) -> concepts-messaging.md doesn't have anchor: #non-persistent-topics.md
[`journalMaxSizeMB`](reference-configuration.md#bookkeeper) -> reference-configuration.md not exist

admin-api-brokers.md has 3 errors:

[Configuration](reference-configuration.md#broker) -> reference-configuration.md not exist
[configuration](reference-configuration.md#broker) -> reference-configuration.md not exist
[`brokerShutdownTimeoutMs`](reference-configuration.md#broker-brokerShutdownTimeoutMs) -> reference-configuration.md not exist

administration-proxy.md has 1 errors:

[Pulsar proxy](reference-configuration.md#pulsar-proxy) -> reference-configuration.md not exist

cookbooks-retention-expiry.md has 1 errors:

[`broker.conf`](reference-configuration.md#broker) -> reference-configuration.md not exist

deploy-bare-metal.md has 11 errors:

[broker configuration](reference-configuration.md#broker) -> reference-configuration.md not exist
[ZooKeeper configuration](reference-configuration.md#zookeeper) -> reference-configuration.md not exist
[`conf/zookeeper.conf`](reference-configuration.md#zookeeper) -> reference-configuration.md not exist
[`dataDir`](reference-configuration.md#zookeeper-dataDir) -> reference-configuration.md not exist
[`conf/bookkeeper.conf`](reference-configuration.md#bookkeeper) -> reference-configuration.md not exist
[`zkServers`](reference-configuration.md#bookkeeper-zkServers) -> reference-configuration.md not exist
[here](reference-configuration.md#bookkeeper) -> reference-configuration.md not exist
[`metadataStoreUrl`](reference-configuration.md#broker) -> reference-configuration.md not exist
[`configurationMetadataStoreUrl`](reference-configuration.md#broker) -> reference-configuration.md not exist
[`conf/broker.conf`](reference-configuration.md#broker) -> reference-configuration.md not exist
[`conf/client.conf`](reference-configuration.md#client) -> reference-configuration.md not exist

reference-metrics.md has 4 errors:

[Request metrics](#request-metrics) -> reference-metrics.md doesn't have anchor: #request-metrics
[Server metrics](#server-metrics-1) -> reference-metrics.md doesn't have anchor: #server-metrics-1
[Replication metrics](#replication-metrics-1) -> reference-metrics.md doesn't have anchor: #replication-metrics-1
[Connection metrics](#connection-metrics) -> reference-metrics.md doesn't have anchor: #connection-metrics

functions-concepts.md has 1 errors:

[cluster-mode](functions-deploy.md#depoy-a-function-in-cluster-mode) -> functions-deploy.md doesn't have anchor: #depoy-a-function-in-cluster-mode

client-libraries-java.md has 3 errors:

[above](#client-configuration) -> client-libraries-java.md doesn't have anchor: #client-configuration
[partitioned topics](concepts-architecture-overview.md#partitioned-topics) -> concepts-architecture-overview.md doesn't have anchor: #partitioned-topics
[above](#client-configuration) -> client-libraries-java.md doesn't have anchor: #client-configuration

concepts-messaging.md has 2 errors:

[configure](reference-configuration.md#broker) -> reference-configuration.md not exist
[configuration](reference-configuration.md#broker-enableNonPersistentTopics) -> reference-configuration.md not exist

admin-api-overview.md has 1 errors:

[`conf/client.conf`](reference-configuration.md#client) -> reference-configuration.md not exist

cookbooks-deduplication.md has 1 errors:

[`broker.conf`](reference-configuration.md#broker) -> reference-configuration.md not exist

develop-schema.md has 1 errors:

[`broker.conf`](reference-configuration.md#broker) -> reference-configuration.md not exist

reference-pulsar-admin.md has 2 errors:

[here](reference-configuration.md#broker) -> reference-configuration.md not exist
[here](reference-configuration.md#broker) -> reference-configuration.md not exist

cookbooks-compaction.md has 1 errors:

[broker configuration](reference-configuration.md#broker) -> reference-configuration.md not exist

admin-api-topics.md has 2 errors:

[here](reference-configuration.md#broker) -> reference-configuration.md not exist
[here](reference-configuration.md#broker) -> reference-configuration.md not exist

deploy-bare-metal-multi-cluster.md has 14 errors:

[broker configuration](reference-configuration.md#broker) -> reference-configuration.md not exist
[ZooKeeper configuration](reference-configuration.md#zookeeper) -> reference-configuration.md not exist
[`conf/zookeeper.conf`](reference-configuration.md#zookeeper) -> reference-configuration.md not exist
[`dataDir`](reference-configuration.md#zookeeper-dataDir) -> reference-configuration.md not exist
[`conf/global_zookeeper.conf`](reference-configuration.md#configuration-store) -> reference-configuration.md not exist
[`conf/bookkeeper.conf`](reference-configuration.md#bookkeeper) -> reference-configuration.md not exist
[`zkServers`](reference-configuration.md#bookkeeper-zkServers) -> reference-configuration.md not exist
[`conf/broker.conf`](reference-configuration.md#broker) -> reference-configuration.md not exist
[`metadataStoreUrl`](reference-configuration.md#broker) -> reference-configuration.md not exist
[`configurationMetadataStoreUrl`](reference-configuration.md#broker) -> reference-configuration.md not exist
[`clusterName`](reference-configuration.md#broker-clusterName) -> reference-configuration.md not exist
[endpoint](reference-configuration.md) -> reference-configuration.md not exist
[`conf/client.conf`](reference-configuration.md#client) -> reference-configuration.md not exist
[`serviceUrl`](reference-configuration.md#client-serviceUrl) -> reference-configuration.md not exist

reference-cli-pulsar-shell.md has 1 errors:

[pulsar-client](#pulsar-client) -> reference-cli-pulsar-shell.md doesn't have anchor: #pulsar-client

[Anonymitaet]

@labuladong FYI: currently, https://pulsar.apache.org/reference rather than https://pulsar.apache.org/tools/ is used (a new change).

[tisonkun]

@Anonymitaet Then we should properly redirect or update the links. Or host the deprecated tools content on an archived source (and properly redirect there).

[labuladong]

@Anonymitaet So I think we can replace all reference-configuration.md with https://pulsar.apache.org/reference. Right?

[tisonkun]

@labuladong not exactly. You may be aware that https://pulsar.apache.org/reference doesn't contain all versions of references. So it's a bit tricky if you globally do the replacement (apply for all versions), you actually cause new issues.

[labuladong]

OK. But all the reference-configuration.md is 404... I think we should do something to optimize it.

[tisonkun]

@labuladong at least for the maintained versions (2.10.x, 2.9.x, 2.8.x, 2.7.x) you can do the replacement. My comments above hint that it can not be applied simply globally - you should check the result instead of making assumptions.

[Anonymitaet]

@labuladong not exactly. You may be aware that https://pulsar.apache.org/reference doesn't contain all versions of references. So it's a bit tricky if you globally do the replacement (apply for all versions), you actually cause new issues.

Exactly