diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-brokers.md b/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-brokers.md index fb353957e0e63..57454d9d57223 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-brokers.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-brokers.md @@ -12,7 +12,7 @@ Pulsar brokers consist of two components: [Brokers](reference-terminology.md#broker) can be managed via: -* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/brokers` endpoint of the admin {@inject: rest:REST:/} API * The `brokers` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin.html} object in the [Java API](client-libraries-java.md) @@ -95,7 +95,7 @@ One way to configure a Pulsar [broker](reference-terminology.md#broker) is to su But since all broker configuration in Pulsar is stored in ZooKeeper, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values. -* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. +* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. * In the Pulsar admin {@inject: rest:REST:/} API, dynamic configuration is managed through the `/admin/v2/brokers/configuration` endpoint. ### Update dynamic configuration diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-clusters.md b/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-clusters.md index c51046cd90619..7b3635de58f35 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-clusters.md @@ -10,7 +10,7 @@ servers (aka [bookies](reference-terminology.md#bookie)), and a [ZooKeeper](http Clusters can be managed via: -* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/clusters` endpoint of the admin {@inject: rest:REST:/} API * The `clusters` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md) @@ -62,7 +62,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > #### No cluster metadata initialization through the REST API or the Java admin API > @@ -86,11 +86,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. #### pulsar-admin diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-namespaces.md b/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-namespaces.md index f07cec7efccc7..33a278de8def1 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-namespaces.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-namespaces.md @@ -9,7 +9,7 @@ Pulsar [namespaces](reference-terminology.md#namespace) are logical groupings of Namespaces can be managed via: -* The [`namespaces`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`namespaces`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/namespaces` endpoint of the admin {@inject: rest:REST:/} API * The `namespaces` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md) diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-overview.md b/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-overview.md index 3d5a05bdaad81..182697cc80325 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-overview.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-overview.md @@ -12,7 +12,7 @@ You can currently interact with the admin interface via: - Making HTTP calls against the admin {@inject: rest:REST:/} API provided by Pulsar [brokers](reference-terminology.md#broker). For some restful apis, they might be redirected to topic owner brokers for serving with [`307 Temporary Redirect`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/307), hence the HTTP callers should handle `307 Temporary Redirect`. If you are using `curl`, you should specify `-L` to handle redirections. -- The `pulsar-admin` CLI tool, which is available in the `bin` folder of your [Pulsar installation](getting-started-standalone): +- The `pulsar-admin` CLI tool, which is available in the `bin` folder of your [Pulsar installation](getting-started-standalone.md): ```shell @@ -20,7 +20,7 @@ $ bin/pulsar-admin ``` -Full documentation for this tool can be found in the [Pulsar command-line tools](reference-pulsar-admin) doc. +Full documentation for this tool can be found in the [Pulsar command-line tools](reference-pulsar-admin.md) doc. - A Java client interface. @@ -31,11 +31,11 @@ In this document, examples from each of the three available interfaces will be s ## Admin setup -Each of Pulsar's three admin interfaces---the [`pulsar-admin`](reference-pulsar-admin) CLI tool, the [Java admin API](/api/admin), and the {@inject: rest:REST:/} API ---requires some special setup if you have [authentication](security-overview.md#authentication-providers) enabled in your Pulsar [instance](reference-terminology.md#instance). +Each of Pulsar's three admin interfaces---the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool, the [Java admin API](/api/admin), and the {@inject: rest:REST:/} API ---requires some special setup if you have [authentication](security-overview.md#authentication-providers) enabled in your Pulsar [instance](reference-terminology.md#instance). ### pulsar-admin -If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: +If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin.md) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: |Name|Description|Default| |----|-----------|-------| diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-partitioned-topics.md b/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-partitioned-topics.md index 0042b8650f87f..4051f494b7a57 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-partitioned-topics.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-partitioned-topics.md @@ -6,7 +6,7 @@ original_id: admin-api-partitioned-topics --- -You can use Pulsar's [admin API](admin-api-overview) to create and manage partitioned topics. +You can use Pulsar's [admin API](admin-api-overview.md) to create and manage partitioned topics. In all of the instructions and commands below, the topic name structure is: diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-schemas.md b/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-schemas.md index ed18490dbdfb0..161f1ee3a3491 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-schemas.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-schemas.md @@ -5,7 +5,7 @@ sidebar_label: "Schemas" original_id: admin-api-schemas --- -Schemas, like other entities in Pulsar, can be managed using the [admin API](admin-api-overview). +Schemas, like other entities in Pulsar, can be managed using the [admin API](admin-api-overview.md). ## Schema resources diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-tenants.md b/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-tenants.md index 924cbce3d41ab..27764d6da3885 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-tenants.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/admin-api-tenants.md @@ -5,7 +5,7 @@ sidebar_label: "Tenants" original_id: admin-api-tenants --- -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/administration-geo.md b/site2/website/versioned_docs/version-2.1.0-incubating/administration-geo.md index bd3cbfa518cc9..2b30491803c9f 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/administration-geo.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/administration-geo.md @@ -94,7 +94,7 @@ Once you've created a global namespace, any topics that producers or consumers c By default, messages are replicated to all clusters configured for the namespace. You can restrict replication selectively by specifying a replication list for a message. That message will then be replicated only to the subset in the replication list. -Below is an example for the [Java API](client-libraries-java). Note the use of the `setReplicationClusters` method when constructing the {@inject: javadoc:Message:/client/org/apache/pulsar/client/api/Message} object: +Below is an example for the [Java API](client-libraries-java.md). Note the use of the `setReplicationClusters` method when constructing the {@inject: javadoc:Message:/client/org/apache/pulsar/client/api/Message} object: ```java @@ -116,7 +116,7 @@ producer.newMessage() #### Topic stats -Topic-specific statistics for global topics are available via the [`pulsar-admin`](reference-pulsar-admin) tool and {@inject: rest:REST:/} API: +Topic-specific statistics for global topics are available via the [`pulsar-admin`](reference-pulsar-admin.md) tool and {@inject: rest:REST:/} API: ```shell diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/administration-proxy.md b/site2/website/versioned_docs/version-2.1.0-incubating/administration-proxy.md index a60d6b9df20d5..c3948e979bebb 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/administration-proxy.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/administration-proxy.md @@ -9,7 +9,7 @@ The [Pulsar proxy](concepts-architecture-overview.md#pulsar-proxy) is an optiona ## Running the proxy -In order to run the Pulsar proxy, you need to have both a local [ZooKeeper](https://zookeeper.apache.org) and configuration store quorum set up for use by your Pulsar cluster. For instructions, see [this document](deploy-bare-metal). Once you have ZooKeeper set up and have connection strings for both ZooKeeper quorums, you can use the [`proxy`](reference-cli-tools.md#pulsar-proxy) command of the [`pulsar`](reference-cli-tools.md#pulsar) CLI tool to start up the proxy (preferably on its own machine or in its own VM): +In order to run the Pulsar proxy, you need to have both a local [ZooKeeper](https://zookeeper.apache.org) and configuration store quorum set up for use by your Pulsar cluster. For instructions, see [this document](deploy-bare-metal.md). Once you have ZooKeeper set up and have connection strings for both ZooKeeper quorums, you can use the [`proxy`](reference-cli-tools.md#pulsar-proxy) command of the [`pulsar`](reference-cli-tools.md#pulsar) CLI tool to start up the proxy (preferably on its own machine or in its own VM): To start the proxy: diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/administration-zk-bk.md b/site2/website/versioned_docs/version-2.1.0-incubating/administration-zk-bk.md index eff0a24f05beb..bdc41b5e07d3d 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/administration-zk-bk.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/administration-zk-bk.md @@ -182,7 +182,7 @@ Configuration for configuration store is handled by the [`conf/global-zookeeper. BookKeeper is responsible for all durable message storage in Pulsar. BookKeeper is a distributed [write-ahead log](https://en.wikipedia.org/wiki/Write-ahead_logging) WAL system that guarantees read consistency of independent message logs called ledgers. Individual BookKeeper servers are also called *bookies*. -> For a guide to managing message persistence, retention, and expiry in Pulsar, see [this cookbook](cookbooks-retention-expiry). +> For a guide to managing message persistence, retention, and expiry in Pulsar, see [this cookbook](cookbooks-retention-expiry.md). ### Deploying BookKeeper diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/client-libraries-go.md b/site2/website/versioned_docs/version-2.1.0-incubating/client-libraries-go.md index 8f3fc7a174ca7..05082c38bfdd6 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/client-libraries-go.md @@ -16,7 +16,7 @@ The Pulsar Go client can be used to create Pulsar [producers](#producers), [cons ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Installing go package @@ -52,7 +52,7 @@ import "github.com/apache/incubator-pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -70,7 +70,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -225,14 +225,14 @@ Parameter | Description | Default :---------|:------------|:------- `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | | `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4) and [`ZLIB`](https://zlib.net/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | ## Consumers @@ -477,7 +477,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/client-libraries-java.md b/site2/website/versioned_docs/version-2.1.0-incubating/client-libraries-java.md index 18961cd46cb9f..c0b6c0762a24f 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/client-libraries-java.md @@ -5,16 +5,16 @@ sidebar_label: "Java" original_id: client-libraries-java --- -The Pulsar Java client can be used both to create Java producers, consumers, and [readers](#readers) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +The Pulsar Java client can be used both to create Java producers, consumers, and [readers](#readers) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. Javadoc for the Pulsar client is divided up into two domains, by package: Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) -This document will focus only on the client API for producing and consuming messages on Pulsar topics. For a guide to using the Java admin client, see [The Pulsar admin interface](admin-api-overview). +This document will focus only on the client API for producing and consuming messages on Pulsar topics. For a guide to using the Java admin client, see [The Pulsar admin interface](admin-api-overview.md). ## Installation @@ -54,7 +54,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -72,7 +72,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -93,7 +93,7 @@ PulsarClient client = PulsarClient.builder() ``` > #### Default broker URLs for standalone clusters -> If you're running a cluster in [standalone mode](getting-started-standalone), the broker will be available at the `pulsar://localhost:6650` URL by default. +> If you're running a cluster in [standalone mode](getting-started-standalone.md), the broker will be available at the `pulsar://localhost:6650` URL by default. Check out the Javadoc for the {@inject: javadoc:PulsarClient:/client/org/apache/pulsar/client/api/PulsarClient} class for a full listing of configurable parameters. @@ -171,7 +171,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -376,7 +376,7 @@ The code sample above shows pointing the `Reader` object to a specific message ( ## Schemas -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](concepts-schema-registry) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. Here's an example: +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](concepts-schema-registry.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. Here's an example: ```java @@ -471,7 +471,7 @@ The following schema formats are currently available for Java: ## Authentication -Pulsar currently supports two authentication schemes: [TLS](security-tls-authentication.md) and [Athenz](security-athenz). The Pulsar Java client can be used with both. +Pulsar currently supports two authentication schemes: [TLS](security-tls-authentication.md) and [Athenz](security-athenz.md). The Pulsar Java client can be used with both. ### TLS Authentication @@ -498,7 +498,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/client-libraries-python.md b/site2/website/versioned_docs/version-2.1.0-incubating/client-libraries-python.md index dc3b22bd1d0d3..013802db1bfc4 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -The Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/incubator-pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +The Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/incubator-pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. ## Installation diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.1.0-incubating/client-libraries-websocket.md index 6a56a8b4487d6..dfc9cbde2f443 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/client-libraries-websocket.md @@ -5,14 +5,14 @@ sidebar_label: "WebSocket" original_id: client-libraries-websocket --- -Pulsar's [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API is meant to provide a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSockets you can publish and consume messages and use all the features available in the [Java](client-libraries-java.md), [Python](client-libraries-python.md), and [C++](client-libraries-cpp) client libraries. +Pulsar's [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API is meant to provide a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSockets you can publish and consume messages and use all the features available in the [Java](client-libraries-java.md), [Python](client-libraries-python.md), and [C++](client-libraries-cpp.md) client libraries. > You can use Pulsar's WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/client-libraries.md b/site2/website/versioned_docs/version-2.1.0-incubating/client-libraries.md index a28b6b3d43419..c8f9d4b83a0a2 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/client-libraries.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/client-libraries.md @@ -14,29 +14,29 @@ Pulsar currently has client libraries available for following languages: ## Java client -For a tutorial on using the Pulsar Java client to produce and consume messages, see [The Pulsar Java client](client-libraries-java). +For a tutorial on using the Pulsar Java client to produce and consume messages, see [The Pulsar Java client](client-libraries-java.md). There are also two independent sets of Javadoc API docs available: Library | Purpose :-------|:------- -[`org.apache.pulsar.client.api`](/api/client) | The [Pulsar Java client](client-libraries-java) for producing and consuming messages on Pulsar topics [`org.apache.pulsar.client.admin`](/api/admin) | The Java client for the [Pulsar admin interface](admin-api-overview.md) +[`org.apache.pulsar.client.api`](/api/client) | The [Pulsar Java client](client-libraries-java.md) for producing and consuming messages on Pulsar topics [`org.apache.pulsar.client.admin`](/api/admin) | The Java client for the [Pulsar admin interface](admin-api-overview.md) ## Go client -For a tutorial on using the Pulsar Go client, see [The Pulsar Go client](client-libraries-go). +For a tutorial on using the Pulsar Go client, see [The Pulsar Go client](client-libraries-go.md). ## Python client -For a tutorial on using the Pulsar Python client, see [The Pulsar Python client](client-libraries-python). +For a tutorial on using the Pulsar Python client, see [The Pulsar Python client](client-libraries-python.md). There are also [pdoc](https://github.com/BurntSushi/pdoc)-generated API docs for the Python client [here](/api/python). ## C++ client -For a tutorial on using the Pulsar C++ clent, see [The Pulsar C++ client](client-libraries-cpp). +For a tutorial on using the Pulsar C++ clent, see [The Pulsar C++ client](client-libraries-cpp.md). There are also [Doxygen](http://www.stack.nl/~dimitri/doxygen/)-generated API docs for the C++ client [here](/api/cpp). diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.1.0-incubating/concepts-architecture-overview.md index 99fdc8de886d2..7dd93f7115eee 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/concepts-architecture-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Architecture" original_id: concepts-architecture-overview --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -17,20 +17,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -40,9 +40,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -130,17 +130,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -148,7 +148,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/concepts-authentication.md b/site2/website/versioned_docs/version-2.1.0-incubating/concepts-authentication.md index 5b94e89e16934..d48e25a1dff9d 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/concepts-authentication.md @@ -5,5 +5,5 @@ sidebar_label: "Authentication and Authorization" original_id: concepts-authentication --- -Pulsar supports a pluggable [authentication](security-overview) mechanism which can be configured at broker and it also supports authorization to identify client and its access rights on topics and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at broker and it also supports authorization to identify client and its access rights on topics and tenants. diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/concepts-clients.md b/site2/website/versioned_docs/version-2.1.0-incubating/concepts-clients.md index 0e8388d950a7a..f3b5ae3893212 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/concepts-clients.md @@ -5,7 +5,7 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md) and [C++](client-libraries-cpp). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md) and [C++](client-libraries-cpp.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/concepts-messaging.md b/site2/website/versioned_docs/version-2.1.0-incubating/concepts-messaging.md index 4049a8b260219..9f66ba5e5c7e1 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/concepts-messaging.md @@ -24,7 +24,7 @@ Publish time | The timestamp of when the message was published (automatically ap Event time | An optional timestamp that applications can attach to the message representing when something happened, e.g. when the message was processed. The event time of a message is 0 if none is explicitly set. -> For a more in-depth breakdown of Pulsar message contents, see the documentation on Pulsar's [binary protocol](developing-binary-protocol). +> For a more in-depth breakdown of Pulsar message contents, see the documentation on Pulsar's [binary protocol](developing-binary-protocol.md). ## Producers @@ -74,7 +74,7 @@ Messages can be acknowledged either one by one or cumulatively. With cumulative ### Listeners -Client libraries can provide their own listener implementations for consumers. The [Java client](client-libraries-java), for example, provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries can provide their own listener implementations for consumers. The [Java client](client-libraries-java.md), for example, provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ## Topics @@ -198,7 +198,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -212,7 +212,7 @@ Key hash | If a key property has been specified on the message, the partitioned Single default partition | If no key is provided, each producer's message will be routed to a dedicated partition, initially random selected | Per-producer ordering Round robin distribution | If no key is provided, all messages will be routed to different partitions in round-robin fashion to achieve maximum throughput. | None -In addition to these default modes, you can also create a custom routing mode if you're using the [Java client](client-libraries-java) by implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +In addition to these default modes, you can also create a custom routing mode if you're using the [Java client](client-libraries-java.md) by implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. @@ -231,7 +231,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -279,7 +279,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -302,12 +302,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, which means that you don't need to modify your Pulsar client code. Instead, you only need to make administrative changes (see the [Managing message deduplication](cookbooks-deduplication) cookbook for a guide). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, which means that you don't need to modify your Pulsar client code. Instead, you only need to make administrative changes (see the [Managing message deduplication](cookbooks-deduplication.md) cookbook for a guide). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.1.0-incubating/concepts-multi-tenancy.md index 685c3031d4aad..8cba09547d2d3 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/concepts-multi-tenancy.md @@ -5,7 +5,7 @@ sidebar_label: "Multi Tenancy" original_id: concepts-multi-tenancy --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -21,7 +21,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -29,7 +29,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/concepts-overview.md b/site2/website/versioned_docs/version-2.1.0-incubating/concepts-overview.md index 743a6f9a47731..02b4ef3d5bb14 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Pulsar's key features include: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters * Very low publish and end-to-end latency * Seamless scalability out to over a million topics * A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Python](client-libraries-python.md), and [C++](client-libraries-cpp.md) * Multiple [subscription types](concepts-messaging.md#subscription-types) for topics ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/) -* A serverless lightweight computing framework [Pulsar Functions](functions-overview) offers stream native data processing. -* A serverless connector framework [Pulsar IO](io-overview) built on-top-of Pulsar Functions to make moving data in and out Apache Pulsar easier. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warn storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless lightweight computing framework [Pulsar Functions](functions-overview.md) offers stream native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md) built on-top-of Pulsar Functions to make moving data in and out Apache Pulsar easier. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warn storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/concepts-replication.md b/site2/website/versioned_docs/version-2.1.0-incubating/concepts-replication.md index 6e23962e99771..799f0eb4d92c6 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/concepts-replication.md @@ -5,5 +5,5 @@ sidebar_label: "Geo Replication" original_id: concepts-replication --- -Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that. +Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo.md) in Pulsar enables you to do that. diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/concepts-schema-registry.md b/site2/website/versioned_docs/version-2.1.0-incubating/concepts-schema-registry.md index 651a3b897a884..d08c0cfa0a99d 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/concepts-schema-registry.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/concepts-schema-registry.md @@ -15,7 +15,7 @@ Both approaches are available in Pulsar, and you're free to adopt one or the oth 1. For the "client-side" approach, producers and consumers can send and receive messages consisting of raw byte arrays and leave all type safety enforcement to the application on an "out-of-band" basis. 1. For the "server-side" approach, Pulsar has a built-in **schema registry** that enables clients to upload data schemas on a per-topic basis. Those schemas dictate which data types are recognized as valid for that topic. -> The Pulsar schema registry is currently available only for the [Java client](client-libraries-java). +> The Pulsar schema registry is currently available only for the [Java client](client-libraries-java.md). ## Basic architecture @@ -37,7 +37,7 @@ Pulsar schemas are fairly simple data structures that consist of: ## Schema versions -In order to illustrate how schema versioning works, let's walk through an example. Imagine that the Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begin sending messages: +In order to illustrate how schema versioning works, let's walk through an example. Imagine that the Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begin sending messages: ```java diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.1.0-incubating/concepts-tiered-storage.md index a0c7abd99acc2..3a67f0949050e 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3 as a long term store. Offloading to S3 triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on bookkeeper, and the broker will copy the backlog data to S3. The original data will then be deleted from bookkeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.1.0-incubating/concepts-topic-compaction.md index e402c965384fd..34b7ed7fbbd31 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/concepts-topic-compaction.md @@ -7,7 +7,7 @@ original_id: concepts-topic-compaction Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -24,7 +24,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.1.0-incubating/cookbooks-compaction.md index d1a3bafffb580..58ee52a622eb7 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually {#trigger} -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.1.0-incubating/cookbooks-non-persistent.md index 8eb1307c3eae5..f831119681634 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/cookbooks-non-persistent.md @@ -41,7 +41,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-non-persistent-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-non-persistent-topics.md) guide. ## Enabling @@ -55,7 +55,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.1.0-incubating/cookbooks-partitioned.md index 2de88249412d3..b033d54790442 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/cookbooks-partitioned.md @@ -7,7 +7,7 @@ original_id: cookbooks-partitioned By default, Pulsar topics are served by a single broker. Using only a single broker, however, limits a topic's maximum throughput. *Partitioned topics* are a special type of topic that can span multiple brokers and thus allow for much higher throughput. For an explanation of how partitioned topics work, see the [Partitioned Topics](concepts-messaging.md#partitioned-topics) concepts. -You can [publish](#publishing-to-partitioned-topics) to partitioned topics using Pulsar's client libraries and you can [create and manage](#managing-partitioned-topics) partitioned topics using Pulsar's [admin API](admin-api-overview). +You can [publish](#publishing-to-partitioned-topics) to partitioned topics using Pulsar's client libraries and you can [create and manage](#managing-partitioned-topics) partitioned topics using Pulsar's [admin API](admin-api-overview.md). ## Publishing to partitioned topics @@ -83,4 +83,4 @@ producer.send("Partitioned topic message".getBytes()); ## Managing partitioned topics -You can use Pulsar's [admin API](admin-api-overview.md) to create and manage [partitioned topics](admin-api-partitioned-topics). +You can use Pulsar's [admin API](admin-api-overview.md) to create and manage [partitioned topics](admin-api-partitioned-topics.md). diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.1.0-incubating/cookbooks-retention-expiry.md index 3d2881fbb87c4..ea2fd385e2e64 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/cookbooks-retention-expiry.md @@ -15,7 +15,7 @@ In Pulsar, you can override both of these default behaviors, at the namespace le * You can persistently store messages that have already been consumed and acknowledged for a minimum time by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe, can be automatically marked as consumed, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL at the namespace level (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL at the namespace level (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL are solving two different problems diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/deploy-aws.md b/site2/website/versioned_docs/version-2.1.0-incubating/deploy-aws.md index f7e0f5490e0ca..488a8de28048b 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/deploy-aws.md @@ -5,7 +5,7 @@ sidebar_label: "Amazon Web Services" original_id: deploy-aws --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary to run the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---while Ansible can install and run Pulsar on the provisioned resources. diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.1.0-incubating/deploy-bare-metal-multi-cluster.md index 1dc7d9e76b7c1..407f04deba583 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/deploy-bare-metal-multi-cluster.md @@ -9,14 +9,14 @@ original_id: deploy-bare-metal-multi-cluster 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you're interested in experimenting with Pulsar or using it in a startup or on a single team, we recommend opting for a single cluster. For instructions on deploying a single cluster, -see the guide [here](deploy-bare-metal). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and make sure it is installed under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). ::: -A Pulsar *instance* consists of multiple Pulsar clusters working in unison. Clusters can be distributed across data centers or geographical regions and can replicate amongst themselves using [geo-replication](administration-geo). Deploying a multi-cluster Pulsar instance involves the following basic steps: +A Pulsar *instance* consists of multiple Pulsar clusters working in unison. Clusters can be distributed across data centers or geographical regions and can replicate amongst themselves using [geo-replication](administration-geo.md). Deploying a multi-cluster Pulsar instance involves the following basic steps: * Deploying two separate [ZooKeeper](#deploying-zookeeper) quorums: a [local](#deploying-local-zookeeper) quorum for each cluster in the instance and a [configuration store](#configuration-store) quorum for instance-wide tasks * Initializing [cluster metadata](#cluster-metadata-initialization) for each cluster @@ -26,7 +26,7 @@ A Pulsar *instance* consists of multiple Pulsar clusters working in unison. Clus If you're deploying a single Pulsar cluster, see the [Clusters and Brokers](getting-started-standalone.md#starting-the-cluster) guide. > #### Running Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes. If you'd like to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you're looking to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes. If you'd like to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you're looking to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). ## System requirement Currently, Pulsar is available for 64-bit **macOS**, **Linux**, and **Windows**. To use Pulsar, you need to install 64-bit JRE/JDK 8 or later versions. @@ -243,7 +243,7 @@ As you can see from the example above, the following needs to be specified: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you're using [TLS](security-tls-transport), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -344,17 +344,17 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). -You can also use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can also use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > #### Service discovery already provided by many scheduling systems -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you're running Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you're running Pulsar on such a system, you may not need to provide your own service discovery mechanism. ### Service discovery setup -The service discovery mechanism included with Pulsar maintains a list of active brokers, stored in ZooKeeper, and supports lookup using HTTP and also Pulsar's [binary protocol](developing-binary-protocol). +The service discovery mechanism included with Pulsar maintains a list of active brokers, stored in ZooKeeper, and supports lookup using HTTP and also Pulsar's [binary protocol](developing-binary-protocol.md). To get started setting up Pulsar's built-in service discovery, you need to change a few parameters in the [`conf/discovery.conf`](reference-configuration.md#service-discovery) configuration file. Set the [`zookeeperServers`](reference-configuration.md#service-discovery-zookeeperServers) parameter to the cluster's ZooKeeper quorum connection string and the [`configurationStoreServers`](reference-configuration.md#service-discovery-configurationStoreServers) setting to the [configuration store](reference-terminology.md#configuration-store) quorum connection string. @@ -379,7 +379,7 @@ $ bin/pulsar-daemon start discovery ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.1.0-incubating/deploy-bare-metal.md index c3dc7b9a377a8..7bc83e0176958 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/deploy-bare-metal.md @@ -9,10 +9,10 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you're interested in experimenting with Pulsar or using it in a startup or on a single team, we recommend opting for a single cluster. If you do need to run a multi-cluster Pulsar instance, -however, see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +however, see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and make sure it is installed under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). ::: @@ -211,9 +211,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (we don't recommend using a different port). -`--web-service-url-tls` | If you're using [TLS](security-tls-transport), you'll also need to specify a TLS web service URL for the cluster. The default port is 8443 (we don't recommend using a different port). +`--web-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster. The default port is 8443 (we don't recommend using a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (we don't recommend using a different port). -`--broker-service-url-tls` | If you're using [TLS](security-tls-transport), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (we don't recommend using a different port). +`--broker-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (we don't recommend using a different port). ## Deploying a BookKeeper cluster @@ -294,7 +294,7 @@ clusterName=pulsar-cluster-1 ### Enabling Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable function worker, by setting `functionsWorkerEnabled` to `true`. diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/deploy-dcos.md b/site2/website/versioned_docs/version-2.1.0-incubating/deploy-dcos.md index 264597a58751e..79919a525c3a7 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -If you want to enable all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of +If you want to enable all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.1.0-incubating/deploy-kubernetes.md index 58446cf789ab7..0684dee963397 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/deploy-kubernetes.md @@ -7,7 +7,7 @@ original_id: deploy-kubernetes :::tip -If you want to enable all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of +If you want to enable all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -20,7 +20,7 @@ The deployment method shown in this guide relies on [YAML](http://yaml.org/) def * A three-node ZooKeeper cluster * A three-broker Pulsar cluster * A [monitoring stack](#) consisting of [Prometheus](https://prometheus.io/), [Grafana](https://grafana.com), and the [Pulsar dashboard](administration-dashboard.md) -* A [pod](https://kubernetes.io/docs/concepts/workloads/pods/pod/) from which you can run administrative commands using the [`pulsar-admin`](reference-pulsar-admin) CLI tool +* A [pod](https://kubernetes.io/docs/concepts/workloads/pods/pod/) from which you can run administrative commands using the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool ## Setup @@ -252,7 +252,7 @@ $ bin/pulsar initialize-cluster-metadata \ ### Deploy the rest of the components -Once cluster metadata has been successfully initialized, you can then deploy the bookies, brokers, monitoring stack ([Prometheus](https://prometheus.io), [Grafana](https://grafana.com), and the [Pulsar dashboard](administration-dashboard)), and Pulsar cluster proxy: +Once cluster metadata has been successfully initialized, you can then deploy the bookies, brokers, monitoring stack ([Prometheus](https://prometheus.io), [Grafana](https://grafana.com), and the [Pulsar dashboard](administration-dashboard.md)), and Pulsar cluster proxy: ```bash @@ -276,9 +276,9 @@ $ kubectl get pods -w -l app=pulsar Once all of the components are up and running, you'll need to create at least one Pulsar tenant and at least one namespace. -> This step is not strictly required if Pulsar [authentication and authorization](security-overview.md) is turned on, though it allows you to change [policies](admin-api-namespaces) for each of the namespaces later. +> This step is not strictly required if Pulsar [authentication and authorization](security-overview.md) is turned on, though it allows you to change [policies](admin-api-namespaces.md) for each of the namespaces later. -You can create properties and namespaces (and perform any other administrative tasks) using the `pulsar-admin` pod that is already configured to act as an admin client for your newly created Pulsar cluster. One easy way to perform administrative tasks is to create an alias for the [`pulsar-admin`](reference-pulsar-admin) tool installed on the admin pod. +You can create properties and namespaces (and perform any other administrative tasks) using the `pulsar-admin` pod that is already configured to act as an admin client for your newly created Pulsar cluster. One easy way to perform administrative tasks is to create an alias for the [`pulsar-admin`](reference-pulsar-admin.md) tool installed on the admin pod. ```bash @@ -349,7 +349,7 @@ $ pulsar-perf consume persistent://public/default/my-topic \ ``` -You can also view [stats](administration-stats) for the topic using the [`pulsar-admin`](reference-pulsar-admin.md#persistent-stats) tool: +You can also view [stats](administration-stats.md) for the topic using the [`pulsar-admin`](reference-pulsar-admin.md#persistent-stats) tool: ```bash @@ -359,7 +359,7 @@ $ pulsar-admin persistent stats persistent://public/default/my-topic ### Monitoring -The default monitoring stack for Pulsar on Kubernetes has consists of [Prometheus](#prometheus), [Grafana](#grafana), and the [Pulsar dashboard](administration-dashboard). +The default monitoring stack for Pulsar on Kubernetes has consists of [Prometheus](#prometheus), [Grafana](#grafana), and the [Pulsar dashboard](administration-dashboard.md). > If you deployed the cluster to Minikube, the following monitoring ports are mapped at the minikube VM: > @@ -389,7 +389,7 @@ You can then access the dashboard in your web browser at [localhost:3000](http:/ #### Pulsar dashboard -While Grafana and Prometheus are used to provide graphs with historical data, [Pulsar dashboard](administration-dashboard) reports more detailed current data for individual [topics](reference-terminology.md#topic). +While Grafana and Prometheus are used to provide graphs with historical data, [Pulsar dashboard](administration-dashboard.md) reports more detailed current data for individual [topics](reference-terminology.md#topic). For example, you can have sortable tables showing all namespaces, topics, and broker stats, with details on the IP address for consumers, how long they've been connected, and much more. diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/deploy-monitoring.md b/site2/website/versioned_docs/version-2.1.0-incubating/deploy-monitoring.md index 8c537e5b1ce72..7a58fa2e3b89a 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/deploy-monitoring.md @@ -89,7 +89,7 @@ For that reason we only collect time series of metrics aggregated at the namespa ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Dashboard](administration-dashboard). +The per-topic dashboard instructions are available at [Dashboard](administration-dashboard.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/develop-schema.md b/site2/website/versioned_docs/version-2.1.0-incubating/develop-schema.md index 42a6a69836774..6940438390ef9 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/develop-schema.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/functions-api.md b/site2/website/versioned_docs/version-2.1.0-incubating/functions-api.md index 4048013937536..36f8586e88b0a 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/functions-api.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/functions-api.md @@ -5,9 +5,9 @@ sidebar_label: "API" original_id: functions-api --- -[Pulsar Functions](functions-overview) provides an easy-to-use API that developers can use to create and manage processing logic for the Apache Pulsar messaging system. With Pulsar Functions, you can write functions of any level of complexity in [Java](#functions-for-java) or [Python](#functions-for-python) and run them in conjunction with a Pulsar cluster without needing to run a separate stream processing engine. +[Pulsar Functions](functions-overview.md) provides an easy-to-use API that developers can use to create and manage processing logic for the Apache Pulsar messaging system. With Pulsar Functions, you can write functions of any level of complexity in [Java](#functions-for-java) or [Python](#functions-for-python) and run them in conjunction with a Pulsar cluster without needing to run a separate stream processing engine. -> For a more in-depth overview of the Pulsar Functions feature, see the [Pulsar Functions overview](functions-overview). +> For a more in-depth overview of the Pulsar Functions feature, see the [Pulsar Functions overview](functions-overview.md). ## Core programming model @@ -45,7 +45,7 @@ Some things to note about this Pulsar Function: ### Example deployment -Deploying Pulsar Functions is handled by the [`pulsar-admin`](reference-pulsar-admin) CLI tool, in particular the [`functions`](reference-pulsar-admin.md#functions) command. Here's an example command that would run our [sanitizer](#example-function) function from above in [local run](functions-deploying.md#local-run-mode) mode: +Deploying Pulsar Functions is handled by the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool, in particular the [`functions`](reference-pulsar-admin.md#functions) command. Here's an example command that would run our [sanitizer](#example-function) function from above in [local run](functions-deploying.md#local-run-mode) mode: ```bash @@ -61,7 +61,7 @@ $ bin/pulsar-admin functions localrun \ ``` -For instructions on running functions in your Pulsar cluster, see the [Deploying Pulsar Functions](functions-deploying) guide. +For instructions on running functions in your Pulsar cluster, see the [Deploying Pulsar Functions](functions-deploying.md) guide. ### Available APIs @@ -542,7 +542,7 @@ public class MetricRecorderFunction implements Function { ``` -> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring) guide. +> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring.md) guide. ## Functions for Python @@ -554,7 +554,7 @@ Writing Pulsar Functions in Python entails implementing one of two things: ### Getting started -Regardless of which [deployment mode](functions-deploying) you're using, you'll need to install the following Python libraries on any machine that's running Pulsar Functions written in Python: +Regardless of which [deployment mode](functions-deploying.md) you're using, you'll need to install the following Python libraries on any machine that's running Pulsar Functions written in Python: * pulsar-client * protobuf diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/functions-deploying.md b/site2/website/versioned_docs/version-2.1.0-incubating/functions-deploying.md index 28276e9deffb3..dfc6dd6795dd2 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/functions-deploying.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/functions-deploying.md @@ -19,8 +19,8 @@ Cluster mode | The function runs *inside of* your Pulsar cluster, on the same ma In order to deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more If you're running a non-[standalone](reference-terminology.md#standalone) cluster, you'll need to obtain the service URL for the cluster. How you obtain the service URL will depend on how you deployed your Pulsar cluster. @@ -52,7 +52,7 @@ Function name | Whichever value is specified for the class name (minus org, libr Tenant | Derived from the input topics' names. If the input topics are under the `marketing` tenant---i.e. the topic names have the form `persistent://marketing/{namespace}/{topicName}`---then the tenant will be `marketing`. Namespace | Derived from the input topics' names. If the input topics are under the `asia` namespace under the `marketing` tenant---i.e. the topic names have the form `persistent://marketing/asia/{topicName}`, then the namespace will be `asia`. Output topic | `{input topic}-{function name}-output`. A function with an input topic name of `incoming` and a function name of `exclamation`, for example, would have an output topic of `incoming-exclamation-output`. -Subscription type | For at-least-once and at-most-once [processing guarantees](functions-guarantees), the [`SHARED`](concepts-messaging.md#shared) is applied by default; for effectively-once guarantees, [`FAILOVER`](concepts-messaging.md#failover) is applied +Subscription type | For at-least-once and at-most-once [processing guarantees](functions-guarantees.md), the [`SHARED`](concepts-messaging.md#shared) is applied by default; for effectively-once guarantees, [`FAILOVER`](concepts-messaging.md#failover) is applied Processing guarantees | [`ATLEAST_ONCE`](functions-guarantees.md) Pulsar service URL | `pulsar://localhost:6650` diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/functions-metrics.md b/site2/website/versioned_docs/version-2.1.0-incubating/functions-metrics.md index 05640f2936f7e..51090086b573f 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/functions-metrics.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/functions-metrics.md @@ -12,7 +12,7 @@ Pulsar Functions can publish arbitrary metrics to the metrics interface which ca ## Accessing metrics -For a guide to accessing metrics created by Pulsar Functions, see the guide to [Monitoring](deploy-monitoring) in Pulsar. +For a guide to accessing metrics created by Pulsar Functions, see the guide to [Monitoring](deploy-monitoring.md) in Pulsar. ## Java SDK diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/functions-overview.md b/site2/website/versioned_docs/version-2.1.0-incubating/functions-overview.md index bb2246a764b62..5dbc04204c2b5 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/functions-overview.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/functions-overview.md @@ -154,7 +154,7 @@ class RoutingFunction(Function): ## Command-line interface -Pulsar Functions are managed using the [`pulsar-admin`](reference-pulsar-admin) CLI tool (in particular the [`functions`](reference-pulsar-admin.md#functions) command). Here's an example command that would run a function in [local run mode](#local-run-mode): +Pulsar Functions are managed using the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool (in particular the [`functions`](reference-pulsar-admin.md#functions) command). Here's an example command that would run a function in [local run mode](#local-run-mode): ```bash @@ -492,7 +492,7 @@ $ bin/pulsar-admin functions create \ ## Metrics -Pulsar Functions that use the [Pulsar Functions SDK](#the-pulsar-functions-sdk) can publish metrics to Pulsar. For more information, see [Metrics for Pulsar Functions](functions-metrics). +Pulsar Functions that use the [Pulsar Functions SDK](#the-pulsar-functions-sdk) can publish metrics to Pulsar. For more information, see [Metrics for Pulsar Functions](functions-metrics.md). ## State storage diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/getting-started-docker.md b/site2/website/versioned_docs/version-2.1.0-incubating/getting-started-docker.md index df1b6b31b37af..0c3d906873356 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/getting-started-docker.md @@ -61,8 +61,8 @@ For more info, see [Topics](concepts-messaging.md#topics). ## Start publishing and consuming messages -Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Python](client-libraries-python), -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Python](client-libraries-python.md), +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: * `pulsar://localhost:6650` @@ -117,7 +117,7 @@ client.close() ## Get the topic statistics In Pulsar you can use REST, Java, or command-line tools to control every aspect of the system. -You can find detailed documentation of all the APIs in the [Admin API Overview](admin-api-overview). +You can find detailed documentation of all the APIs in the [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/getting-started-standalone.md b/site2/website/versioned_docs/version-2.1.0-incubating/getting-started-standalone.md index c73b6dd0d12e9..ed672efe71977 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For the purposes of local development and testing, you can run Pulsar in standalone mode on your own machine. Standalone mode includes a Pulsar broker as well as the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Run Pulsar Standalone Manually @@ -113,7 +113,7 @@ pulsar-io-twitter-@pulsar:version@.nar > If you are running Pulsar in a bare metal cluster, you need to make sure `connectors` tarball is unzipped in every broker's pulsar directory > (or in every function-worker's pulsar directory if you are running a separate worker cluster for Pulsar functions). > -> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), > you can use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ### Starting the cluster @@ -162,12 +162,12 @@ If the message has been successfully published to the topic, you should see a co ## Using Pulsar clients locally -Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Python](client-libraries-python.md), and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: +Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Python](client-libraries-python.md), and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: * `http://localhost:8080` * `pulsar://localhost:6650` -Here's an example producer for a Pulsar topic using the [Java](client-libraries-java) client: +Here's an example producer for a Pulsar topic using the [Java](client-libraries-java.md) client: ```java @@ -178,7 +178,7 @@ Producer producer = client.newProducer().topic("my-topic").create(); ``` -Here's an example [Python](client-libraries-python) producer: +Here's an example [Python](client-libraries-python.md) producer: ```python @@ -189,7 +189,7 @@ producer = client.create_producer('my-topic') ``` -Finally, here's an example [C++](client-libraries-cpp) producer: +Finally, here's an example [C++](client-libraries-cpp.md) producer: ```cpp diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/io-cassandra.md b/site2/website/versioned_docs/version-2.1.0-incubating/io-cassandra.md index 9c46f73da3718..68af2c066e994 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/io-cassandra.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/io-cassandra.md @@ -7,7 +7,7 @@ original_id: io-cassandra The Cassandra Sink connector is used to write messages to a Cassandra Cluster. -The tutorial [Connecting Pulsar with Apache Cassandra](io-quickstart) shows an example how to use Cassandra Sink +The tutorial [Connecting Pulsar with Apache Cassandra](io-quickstart.md) shows an example how to use Cassandra Sink connector to write messages to a Cassandra table. ## Sink Configuration Options diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/io-develop.md b/site2/website/versioned_docs/version-2.1.0-incubating/io-develop.md index b9d211af4228b..a329104201597 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/io-develop.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/io-develop.md @@ -8,12 +8,12 @@ original_id: io-develop This guide describes how developers can write new connectors for Pulsar IO to move data between Pulsar and other systems. It describes how to create a Pulsar IO connector. -Pulsar IO connectors are specialized [Pulsar Functions](functions-overview). So writing +Pulsar IO connectors are specialized [Pulsar Functions](functions-overview.md). So writing a Pulsar IO connector is as simple as writing a Pulsar function. Pulsar IO connectors come in two flavors: {@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}, which import data from another system, and {@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}, -which export data to another system. For example, [KinesisSink](io-kinesis) would export -the messages of a Pulsar topic to a Kinesis stream, and [RabbitmqSource](io-rabbitmq) would import +which export data to another system. For example, [KinesisSink](io-kinesis.md) would export +the messages of a Pulsar topic to a Kinesis stream, and [RabbitmqSource](io-rabbitmq.md) would import the messages of a RabbitMQ queue to a Pulsar topic. ### Developing @@ -137,7 +137,7 @@ separate integration tests to verify end-to-end functionality. In Pulsar, we are ### Packaging Once you've developed and tested your connector, you must package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. There are two approaches described +to a [Pulsar Functions](functions-overview.md) cluster. There are two approaches described here work with Pulsar Functions' runtime. If you plan to package and distribute your connector for others to use, you are obligated to @@ -153,7 +153,7 @@ The easiest approach to packaging a Pulsar IO connector is to create a NAR packa NAR stands for NiFi Archive. It is a custom packaging mechanism used by Apache NiFi, to provide a bit of Java ClassLoader isolation. For more details, you can read this [blog post](https://medium.com/hashmapinc/nifi-nar-files-explained-14113f7796fd) to understand -how NAR works. Pulsar uses the same mechanism for packaging all the [builtin connectors](io-connectors). +how NAR works. Pulsar uses the same mechanism for packaging all the [builtin connectors](io-connectors.md). All what you need is to include this [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin) in your maven project for your connector. For example: diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/io-managing.md b/site2/website/versioned_docs/version-2.1.0-incubating/io-managing.md index c5c61cc4db7ea..c25e15641efb8 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/io-managing.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/io-managing.md @@ -50,7 +50,7 @@ For details, consult the documentation for [individual connectors](io-overview.m ## Running Connectors -Pulsar connectors can be managed using the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands of the [`pulsar-admin`](reference-pulsar-admin) CLI tool. +Pulsar connectors can be managed using the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands of the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. ### Running sources @@ -166,8 +166,8 @@ Here's an example to submit a Cassandra sink: ## Monitoring Connectors -Since Pulsar IO connectors are running as [Pulsar Functions](functions-overview), so you can use [`functions`](reference-pulsar-admin.md#source) commands -available in the [`pulsar-admin`](reference-pulsar-admin) CLI tool. +Since Pulsar IO connectors are running as [Pulsar Functions](functions-overview.md), so you can use [`functions`](reference-pulsar-admin.md#source) commands +available in the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. ### Retrieve Connector Metadata diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/io-overview.md b/site2/website/versioned_docs/version-2.1.0-incubating/io-overview.md index e0f2277e18ec2..7479e1cf5ff6d 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/io-overview.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/io-overview.md @@ -8,7 +8,7 @@ original_id: io-overview Messaging systems are most powerful when you can easily use them in conjunction with external systems like databases and other messaging systems. **Pulsar IO** is a feature of Pulsar that enables you to easily create, deploy, and manage Pulsar **connectors** that interact with external systems, such as [Apache Cassandra](https://cassandra.apache.org), [Aerospike](https://www.aerospike.com), and many others. > #### Pulsar IO and Pulsar Functions -> Under the hood, Pulsar IO connectors are specialized [Pulsar Functions](functions-overview.md) purpose-built to interface with external systems. The [administrative interface](io-quickstart) for Pulsar IO is, in fact, quite similar to that of Pulsar Functions. +> Under the hood, Pulsar IO connectors are specialized [Pulsar Functions](functions-overview.md) purpose-built to interface with external systems. The [administrative interface](io-quickstart.md) for Pulsar IO is, in fact, quite similar to that of Pulsar Functions. ## Sources and sinks @@ -23,7 +23,7 @@ This diagram illustrates the relationship between sources, sinks, and Pulsar: ## Working with connectors -Pulsar IO connectors can be managed via the [`pulsar-admin`](reference-pulsar-admin) CLI tool, in particular the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands. +Pulsar IO connectors can be managed via the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool, in particular the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands. > For a guide to managing connectors in your Pulsar installation, see the [Getting started with Pulsar IO](io-quickstart.md) @@ -31,8 +31,8 @@ The following connectors are currently available for Pulsar: |Name|Java Class|Documentation| |---|---|---| -|[Aerospike sink](https://www.aerospike.com/)|[`org.apache.pulsar.io.aerospike.AerospikeSink`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-io/aerospike/src/main/java/org/apache/pulsar/io/aerospike/AerospikeStringSink.java)|[Documentation](io-aerospike)| -|[Cassandra sink](https://cassandra.apache.org)|[`org.apache.pulsar.io.cassandra.CassandraSink`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-io/cassandra/src/main/java/org/apache/pulsar/io/cassandra/CassandraStringSink.java)|[Documentation](io-cassandra)| +|[Aerospike sink](https://www.aerospike.com/)|[`org.apache.pulsar.io.aerospike.AerospikeSink`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-io/aerospike/src/main/java/org/apache/pulsar/io/aerospike/AerospikeStringSink.java)|[Documentation](io-aerospike.md)| +|[Cassandra sink](https://cassandra.apache.org)|[`org.apache.pulsar.io.cassandra.CassandraSink`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-io/cassandra/src/main/java/org/apache/pulsar/io/cassandra/CassandraStringSink.java)|[Documentation](io-cassandra.md)| |[Kafka source](https://kafka.apache.org)|[`org.apache.pulsar.io.kafka.KafkaSource`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaStringSource.java)|[Documentation](io-kafka.md#source)| |[Kafka sink](https://kafka.apache.org)|[`org.apache.pulsar.io.kafka.KafkaSink`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaStringSink.java)|[Documentation](io-kafka.md#sink)| |[Kinesis sink](https://aws.amazon.com/kinesis/)|[`org.apache.pulsar.io.kinesis.KinesisSink`](https://github.com/apache/incubator-pulsar/blob/master/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java)|[Documentation](io-kinesis.md#sink)| diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/io-quickstart.md b/site2/website/versioned_docs/version-2.1.0-incubating/io-quickstart.md index 36e11e280d351..75031e202edac 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/io-quickstart.md @@ -6,14 +6,14 @@ original_id: io-quickstart --- This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O in tandem with running the steps in this guide +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O in tandem with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you will be able to: - Connect your Pulsar cluster with your Cassandra cluster :::tip -1. These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However all +1. These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However all the commands used in this tutorial should be able to be used in a multi-nodes Pulsar cluster without any changes. 2. All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -267,11 +267,11 @@ configs: ``` -To learn more about Cassandra Connector, see [Cassandra Connector](io-cassandra). +To learn more about Cassandra Connector, see [Cassandra Connector](io-cassandra.md). ### Submit a Cassandra Sink -Pulsar provides the [CLI](reference-cli-tools) for running and managing Pulsar I/O connectors. +Pulsar provides the [CLI](reference-cli-tools.md) for running and managing Pulsar I/O connectors. We can run following command to sink a sink connector with type `cassandra` and config file `examples/cassandra-sink.yml`. @@ -296,7 +296,7 @@ as a Pulsar Function and write the messages produced in topic `test_cassandra` t ### Inspect the Cassandra Sink -Since an IO connector is running as [Pulsar Functions](functions-overview), you can use [functions CLI](reference-pulsar-admin.md#functions) +Since an IO connector is running as [Pulsar Functions](functions-overview.md), you can use [functions CLI](reference-pulsar-admin.md#functions) for inspecting and managing the IO connectors. #### Retrieve Sink Info diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/pulsar-2.0.md b/site2/website/versioned_docs/version-2.1.0-incubating/pulsar-2.0.md index 6a0246562c0cf..d92cf9c7b52a0 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/pulsar-2.0.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/pulsar-2.0.md @@ -5,13 +5,13 @@ sidebar_label: "Pulsar 2.0" original_id: pulsar-2.0 --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/reference-cli-tools.md b/site2/website/versioned_docs/version-2.1.0-incubating/reference-cli-tools.md index 31b6a34381421..4ae70b77834c5 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/security-athenz.md b/site2/website/versioned_docs/version-2.1.0-incubating/security-athenz.md index f7c7e68b0fbad..1f9a0bb694c2a 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/security-athenz.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/security-athenz.md @@ -76,7 +76,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You’ll need to add the following authentication parameters to that file to use Athenz with Pulsar’s CLI tools: diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/security-authorization.md b/site2/website/versioned_docs/version-2.1.0-incubating/security-authorization.md index 890361fc2da6f..e2025ad6b2727 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/security-authorization.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/security-authorization.md @@ -29,7 +29,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters available in the `conf/broker.conf` file, > as well as the default values for those parameters, can be found in [Broker Configuration](reference-configuration.md#broker) -Typically, superuser roles are used for administrators and clients but also for broker-to-broker authorization. When using [geo-replication](concepts-replication), every broker needs to be able to publish to all the other clusters' topics. +Typically, superuser roles are used for administrators and clients but also for broker-to-broker authorization. When using [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other clusters' topics. ### Proxy Roles @@ -60,7 +60,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role A Pulsar [tenant](reference-terminology.md#tenant) is typically provisioned by Pulsar [instance](reference-terminology.md#instance) administrators or by some kind of self-service portal. -Tenants are managed using the [`pulsar-admin`](reference-pulsar-admin) tool. Here's an example tenant creation command: +Tenants are managed using the [`pulsar-admin`](reference-pulsar-admin.md) tool. Here's an example tenant creation command: ```shell @@ -84,7 +84,7 @@ persistent://tenant/namespace/topic ### Managing permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/security-extending.md b/site2/website/versioned_docs/version-2.1.0-incubating/security-extending.md index 9fcbc4f4cebf0..0c30105616ec9 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/security-extending.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/security-extending.md @@ -10,7 +10,7 @@ Pulsar provides a way to use custom authentication and authorization mechanisms ## Authentication Pulsar support mutual TLS and Athenz authentication plugins, and these can be used as described -in [Security](security-overview). +in [Security](security-overview.md). It is possible to use a custom authentication mechanism by providing the implementation in the form of two plugins one for the Client library and the other for the Pulsar Broker to validate diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/security-overview.md b/site2/website/versioned_docs/version-2.1.0-incubating/security-overview.md index 62d51c45c0bbf..6507a93733990 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/security-overview.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/security-overview.md @@ -22,7 +22,7 @@ In Pulsar, a *role* is a string, like `admin` or `app1`, that can represent a si to produce or consume from certain topics, administer the configuration for tenants, and more. Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign that client a *role token*. This -role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication Providers @@ -33,7 +33,7 @@ Currently Pulsar supports two authentication providers: ## Contents -- [Encryption](security-tls-transport.md) and [Authentication](security-tls-authentication) using TLS +- [Encryption](security-tls-transport.md) and [Authentication](security-tls-authentication.md) using TLS - [Authentication using Athenz](security-athenz.md) - [Authorization and ACLs](security-authorization.md) - [End-to-End Encryption](security-encryption.md) diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/security-tls-authentication.md b/site2/website/versioned_docs/version-2.1.0-incubating/security-tls-authentication.md index 3a4fdd7fb1aca..5d2f8dcf5a419 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/security-tls-authentication.md @@ -7,7 +7,7 @@ original_id: security-tls-authentication ## TLS Authentication Overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport), but instead of only servers having keys and certs which the client uses the verify the server's identity, clients also have keys and certs which the server uses to verify the client's identity. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md), but instead of only servers having keys and certs which the client uses the verify the server's identity, clients also have keys and certs which the server uses to verify the client's identity. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. ### Creating client certificates @@ -80,7 +80,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, put the following in `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. The role token for this key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. The role token for this key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -100,7 +100,7 @@ When TLS authentication, the client needs to connect via TLS transport, so you n ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You'll need to add the following parameters to that file to use TLS authentication with Pulsar's CLI tools: diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/security-tls-transport.md b/site2/website/versioned_docs/version-2.1.0-incubating/security-tls-transport.md index bde940cfbe3d2..e08511981015a 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text, which means that all data is sent in the clear. TLS can be used to encrypt this traffic so that it cannot be snooped by a man-in-the-middle attacker. -TLS can be configured for both encryption and authentication. You may configure just TLS transport encryption, which is covered in this guide. TLS authentication is covered [elsewhere](security-tls-authentication.md). Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +TLS can be configured for both encryption and authentication. You may configure just TLS transport encryption, which is covered in this guide. TLS authentication is covered [elsewhere](security-tls-authentication.md). Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may have a performance impact due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -A third kind of key pair, **client key pairs**, are used for [client authentication](security-tls-authentication). +A third kind of key pair, **client key pairs**, are used for [client authentication](security-tls-authentication.md). The **certificate authority** private key should be stored in a very secure location (a fully encrypted, disconnected, air gapped computer). The certificate authority public key, the **trust cert**, can be freely shared. @@ -27,7 +27,7 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server they are talking to has a key pair that was signed by the certificate authority. A man-in-the-middle attacker would not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that was signed by the certificate authority. The Common Name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that was signed by the certificate authority. The Common Name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). ## Creating TLS Certificates @@ -127,7 +127,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Broker Configuration -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you'll need to make some changes to `broker.conf`, which is located in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you'll need to make some changes to `broker.conf`, which is located in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -169,7 +169,7 @@ As the server certificate you generated above doesn't belong to any of the defau ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You'll need to add the following parameters to that file to use TLS transport with Pulsar's CLI tools: diff --git a/site2/website/versioned_docs/version-2.1.0-incubating/standalone.md b/site2/website/versioned_docs/version-2.1.0-incubating/standalone.md index 244f0b3ea720e..73e351cf79a75 100644 --- a/site2/website/versioned_docs/version-2.1.0-incubating/standalone.md +++ b/site2/website/versioned_docs/version-2.1.0-incubating/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For the purposes of local development and testing, you can run Pulsar in standalone mode on your own machine. Standalone mode includes a Pulsar broker as well as the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Run Pulsar Standalone Manually @@ -113,7 +113,7 @@ pulsar-io-twitter-@pulsar:version@.nar > If you are running Pulsar in a bare metal cluster, you need to make sure `connectors` tarball is unzipped in every broker's pulsar directory > (or in every function-worker's pulsar directory if you are running a separate worker cluster for Pulsar functions). > -> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), > you can use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ### Starting the cluster @@ -162,12 +162,12 @@ If the message has been successfully published to the topic, you should see a co ## Using Pulsar clients locally -Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Python](client-libraries-python.md), and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: +Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Python](client-libraries-python.md), and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: * `http://localhost:8080` * `pulsar://localhost:6650` -Here's an example producer for a Pulsar topic using the [Java](client-libraries-java) client: +Here's an example producer for a Pulsar topic using the [Java](client-libraries-java.md) client: ```java @@ -178,7 +178,7 @@ Producer producer = client.newProducer().topic("my-topic").create(); ``` -Here's an example [Python](client-libraries-python) producer: +Here's an example [Python](client-libraries-python.md) producer: ```python @@ -189,7 +189,7 @@ producer = client.create_producer('my-topic') ``` -Finally, here's an example [C++](client-libraries-cpp) producer: +Finally, here's an example [C++](client-libraries-cpp.md) producer: ```cpp diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/admin-api-clusters.md b/site2/website/versioned_docs/version-2.1.1-incubating/admin-api-clusters.md index 04db2f60759bc..1255ad25d7aac 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/admin-api-clusters.md @@ -88,7 +88,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -112,11 +112,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block > - For the latest and complete information about `Java admin API`, including classes, methods, descriptions, and more, see [Java admin API doc](https://pulsar.apache.org/api/admin/). -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -121,7 +121,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries-go.md b/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries-go.md index 33059163f4c6b..d3c2fa5d2fc85 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries-go.md @@ -55,7 +55,7 @@ $ ./test_example ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -81,7 +81,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -1022,7 +1022,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker @@ -1042,7 +1042,7 @@ opts := pulsar.ClientOptions{ ## OAuth2 authentication -To use [OAuth2 authentication](security-oauth2), you'll need to configure your client to perform the following operations. +To use [OAuth2 authentication](security-oauth2.md), you'll need to configure your client to perform the following operations. This example shows how to configure OAuth2 authentication. ```go diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries-java.md b/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries-java.md index 2c349dbf12f12..1058b0896fa78 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries-java.md @@ -10,7 +10,7 @@ import TabItem from '@theme/TabItem'; ```` -You can use a Pulsar Java client to create the Java [producer](#producer), [consumer](#consumer), [reader](#reader) and [TableView](#tableview) of messages and to perform [administrative tasks](admin-api-overview). The current Java client version is **@pulsar:version@**. +You can use a Pulsar Java client to create the Java [producer](#producer), [consumer](#consumer), [reader](#reader) and [TableView](#tableview) of messages and to perform [administrative tasks](admin-api-overview.md). The current Java client version is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), [reader](#reader) and [TableView](#tableview) of a Java client are thread-safe. @@ -19,10 +19,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | [The producer and consumer API](https://pulsar.apache.org/api/client/) | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Include both `pulsar-client` and `pulsar-client-admin`
Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.
In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -69,7 +69,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -95,7 +95,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -126,7 +126,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -279,7 +279,7 @@ Currently, cluster-level failover can perform probes to prevent data loss, but i > #### What are the relationships between cluster-level failover and geo-replication? -The cluster-level failover is an extension of [geo-replication](concepts-replication) to improve stability and robustness. The cluster-level failover depends on geo-replication, and they have some **differences** as below. +The cluster-level failover is an extension of [geo-replication](concepts-replication.md) to improve stability and robustness. The cluster-level failover depends on geo-replication, and they have some **differences** as below. Influence |Cluster-level failover|Geo-replication |---|---|--- @@ -576,7 +576,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics cookbook](cookbooks-partitioned). +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics cookbook](cookbooks-partitioned.md). ### Async send @@ -1370,7 +1370,7 @@ tv.forEach((key, value) -> /*operations on all existing messages*/) ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producer) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producer) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -1488,7 +1488,7 @@ For example of ProtobufNativeSchema, see [`SchemaDefinition` in `Complex type`]( ## Authentication -Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2). You can use the Pulsar Java client with all of them. +Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2.md). You can use the Pulsar Java client with all of them. ### TLS Authentication @@ -1515,7 +1515,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` @@ -1552,7 +1552,7 @@ PulsarClient client = PulsarClient.builder() ### Oauth2 -The following example shows how to use [Oauth2](security-oauth2) as an authentication provider for the Pulsar Java client. +The following example shows how to use [Oauth2](security-oauth2.md) as an authentication provider for the Pulsar Java client. You can use the factory method to configure authentication for Pulsar Java client. diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries-python.md b/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries-python.md index 27584d502a71d..425aa0827d05b 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -The Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/incubator-pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +The Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/incubator-pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. ## Installation diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries-websocket.md index a6e60361749de..0b31ffe78161a 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries-websocket.md @@ -4,14 +4,14 @@ title: Pulsar WebSocket API sidebar_label: "WebSocket" --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries.md b/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries.md index f6b76ab065f88..8ebc0da713893 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/client-libraries.md @@ -14,29 +14,29 @@ Pulsar currently has client libraries available for following languages: ## Java client -For a tutorial on using the Pulsar Java client to produce and consume messages, see [The Pulsar Java client](client-libraries-java). +For a tutorial on using the Pulsar Java client to produce and consume messages, see [The Pulsar Java client](client-libraries-java.md). There are also two independent sets of Javadoc API docs available: Library | Purpose :-------|:------- -[`org.apache.pulsar.client.api`](/api/client) | The [Pulsar Java client](client-libraries-java) for producing and consuming messages on Pulsar topics [`org.apache.pulsar.client.admin`](/api/admin) | The Java client for the [Pulsar admin interface](admin-api-overview.md) +[`org.apache.pulsar.client.api`](/api/client) | The [Pulsar Java client](client-libraries-java.md) for producing and consuming messages on Pulsar topics [`org.apache.pulsar.client.admin`](/api/admin) | The Java client for the [Pulsar admin interface](admin-api-overview.md) ## Go client -For a tutorial on using the Pulsar Go client, see [The Pulsar Go client](client-libraries-go). +For a tutorial on using the Pulsar Go client, see [The Pulsar Go client](client-libraries-go.md). ## Python client -For a tutorial on using the Pulsar Python client, see [The Pulsar Python client](client-libraries-python). +For a tutorial on using the Pulsar Python client, see [The Pulsar Python client](client-libraries-python.md). There are also [pdoc](https://github.com/BurntSushi/pdoc)-generated API docs for the Python client [here](/api/python). ## C++ client -For a tutorial on using the Pulsar C++ clent, see [The Pulsar C++ client](client-libraries-cpp). +For a tutorial on using the Pulsar C++ clent, see [The Pulsar C++ client](client-libraries-cpp.md). There are also [Doxygen](http://www.stack.nl/~dimitri/doxygen/)-generated API docs for the C++ client [here](/api/cpp). diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.1.1-incubating/concepts-architecture-overview.md index 0d96b59f8ae91..1ce346fc16941 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/concepts-architecture-overview.md @@ -4,7 +4,7 @@ title: Architecture Overview sidebar_label: "Architecture" --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -16,20 +16,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers. The producers connect to the brokers to publish messages and the consumers connect to the brokers to consume the messages. -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -39,9 +39,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -139,17 +139,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -157,7 +157,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/concepts-authentication.md b/site2/website/versioned_docs/version-2.1.1-incubating/concepts-authentication.md index 335da8dc5a519..fbefead69f382 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/concepts-authentication.md @@ -4,5 +4,5 @@ title: Authentication and Authorization sidebar_label: "Authentication and Authorization" --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/concepts-clients.md b/site2/website/versioned_docs/version-2.1.1-incubating/concepts-clients.md index 65201b5ce1cd2..c0fe61cd06e76 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/concepts-clients.md @@ -4,12 +4,12 @@ title: Pulsar Clients sidebar_label: "Clients" --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -37,7 +37,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/concepts-messaging.md b/site2/website/versioned_docs/version-2.1.1-incubating/concepts-messaging.md index 89b6827d73d5b..11b15cf814aa5 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/concepts-messaging.md @@ -22,8 +22,8 @@ Messages are the basic "unit" of Pulsar. The following table lists the component Component | Description :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | The key (string type) of the message. It is a short name of message key or partition key. Messages are optionally tagged with keys, which is useful for features like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | The key (string type) of the message. It is a short name of message key or partition key. Messages are optionally tagged with keys, which is useful for features like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer who produces the message. If you do not specify a producer name, the default name is used. Topic name | The name of the topic that the message is published to. @@ -53,7 +53,7 @@ The default size of a message is 5 MB. You can configure the max size of a messa ``` -> For more information on Pulsar messages, see Pulsar [binary protocol](developing-binary-protocol). +> For more information on Pulsar messages, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -171,7 +171,7 @@ Messages are received from [brokers](reference-terminology.md#broker) either syn ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -656,7 +656,7 @@ A [subscription](#subscriptions) can have one or more consumers. When a consumer #### When to use -By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry). +By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry.md). #### How to use @@ -760,7 +760,7 @@ Decisions about routing and subscription modes can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -772,7 +772,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -810,7 +810,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -881,7 +881,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -904,12 +904,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication.md). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.1.1-incubating/concepts-multi-tenancy.md index 8a17e72a4c8b3..26bd6a2fd183f 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/concepts-multi-tenancy.md @@ -4,7 +4,7 @@ title: Multi Tenancy sidebar_label: "Multi Tenancy" --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -20,7 +20,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -28,7 +28,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/concepts-overview.md b/site2/website/versioned_docs/version-2.1.1-incubating/concepts-overview.md index 119d46508ed28..bf4993a3110b5 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/concepts-overview.md @@ -8,15 +8,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/concepts-replication.md b/site2/website/versioned_docs/version-2.1.1-incubating/concepts-replication.md index d4653ca7aecd8..9cc3a7c3415bd 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/concepts-replication.md @@ -8,7 +8,7 @@ Regardless of industries, when an unforeseen event occurs and brings day-to-day Pulsar's geo-replication mechanism is typically used for disaster recovery, enabling the replication of persistently stored message data across multiple data centers. For instance, your application is publishing data in one region and you would like to process it for consumption in other regions. With Pulsar’s geo-replication mechanism, messages can be produced and consumed in different geo-locations. -The diagram below illustrates the process of [geo-replication](administration-geo). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. +The diagram below illustrates the process of [geo-replication](administration-geo.md). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. ![A typical geo-replication example with full-mesh pattern](/assets/full-mesh-replication.svg) diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.1.1-incubating/concepts-tiered-storage.md index f0bbde61718fb..ec031253ccd8d 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/concepts-tiered-storage.md @@ -14,4 +14,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.1.1-incubating/concepts-topic-compaction.md index 790ce76b7c5e0..300cf68956dd1 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/concepts-topic-compaction.md @@ -6,7 +6,7 @@ sidebar_label: "Topic Compaction" Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -23,7 +23,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.1.1-incubating/cookbooks-compaction.md index 0720019aaa035..214e899fa086d 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/cookbooks-compaction.md @@ -44,7 +44,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.1.1-incubating/cookbooks-non-persistent.md index d40c4fbd0cf01..1346d19a04263 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/cookbooks-non-persistent.md @@ -40,7 +40,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -54,7 +54,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.1.1-incubating/cookbooks-partitioned.md index 25896934788fc..c857d019edc02 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/cookbooks-partitioned.md @@ -3,4 +3,4 @@ id: cookbooks-partitioned title: Partitioned topics sidebar_label: "Partitioned Topics" --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.1.1-incubating/cookbooks-retention-expiry.md index 86160c3f7ecf8..cd764fad6eb37 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/cookbooks-retention-expiry.md @@ -21,7 +21,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/deploy-aws.md b/site2/website/versioned_docs/version-2.1.1-incubating/deploy-aws.md index 5db846fe790f2..60ac4c22d7672 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/deploy-aws.md @@ -4,7 +4,7 @@ title: Deploying a Pulsar cluster on AWS using Terraform and Ansible sidebar_label: "Amazon Web Services" --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -209,7 +209,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.1.1-incubating/deploy-bare-metal-multi-cluster.md index 70dff45080b03..8a765d8843f43 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/deploy-bare-metal-multi-cluster.md @@ -6,13 +6,13 @@ sidebar_label: "Bare metal multi-cluster" :::tip -1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster). -2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster.md). +2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo).Deploying a multi-cluster Pulsar instance consists of the following steps: +A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md).Deploying a multi-cluster Pulsar instance consists of the following steps: 1. Deploying two separate ZooKeeper quorums: a local quorum for each cluster in the instance and a configuration store quorum for instance-wide tasks 2. Initializing cluster metadata for each cluster @@ -21,7 +21,7 @@ A Pulsar instance consists of multiple Pulsar clusters working in unison. You ca > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. ## System requirement @@ -65,7 +65,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -244,7 +244,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -363,16 +363,16 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. -You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > **Service discovery already provided by many scheduling systems** -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.1.1-incubating/deploy-bare-metal.md index c3dc7b9a377a8..7bc83e0176958 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/deploy-bare-metal.md @@ -9,10 +9,10 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you're interested in experimenting with Pulsar or using it in a startup or on a single team, we recommend opting for a single cluster. If you do need to run a multi-cluster Pulsar instance, -however, see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +however, see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and make sure it is installed under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). ::: @@ -211,9 +211,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (we don't recommend using a different port). -`--web-service-url-tls` | If you're using [TLS](security-tls-transport), you'll also need to specify a TLS web service URL for the cluster. The default port is 8443 (we don't recommend using a different port). +`--web-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster. The default port is 8443 (we don't recommend using a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (we don't recommend using a different port). -`--broker-service-url-tls` | If you're using [TLS](security-tls-transport), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (we don't recommend using a different port). +`--broker-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (we don't recommend using a different port). ## Deploying a BookKeeper cluster @@ -294,7 +294,7 @@ clusterName=pulsar-cluster-1 ### Enabling Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable function worker, by setting `functionsWorkerEnabled` to `true`. diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.1.1-incubating/deploy-kubernetes.md index 4e170dcd4711d..7455247c901d4 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/deploy-kubernetes.md @@ -5,6 +5,6 @@ sidebar_label: "Kubernetes" --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/deploy-monitoring.md b/site2/website/versioned_docs/version-2.1.1-incubating/deploy-monitoring.md index adf3587b7aeb3..d581fe2edfacb 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/deploy-monitoring.md @@ -112,7 +112,7 @@ http://$FUNCTIONS_WORKER_ADDRESS:$WORKER_PORT/metrics: You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy). +When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy.md). ## Dashboards @@ -120,7 +120,7 @@ When you collect time series statistics, the major problem is to make sure the n ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager). +The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/develop-schema.md b/site2/website/versioned_docs/version-2.1.1-incubating/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/develop-schema.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/functions-api.md b/site2/website/versioned_docs/version-2.1.1-incubating/functions-api.md index ff265f3fc9c34..3c4ffa027368b 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/functions-api.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/functions-api.md @@ -5,9 +5,9 @@ sidebar_label: "API" original_id: functions-api --- -[Pulsar Functions](functions-overview) provides an easy-to-use API that developers can use to create and manage processing logic for the Apache Pulsar messaging system. With Pulsar Functions, you can write functions of any level of complexity in [Java](#functions-for-java) or [Python](#functions-for-python) and run them in conjunction with a Pulsar cluster without needing to run a separate stream processing engine. +[Pulsar Functions](functions-overview.md) provides an easy-to-use API that developers can use to create and manage processing logic for the Apache Pulsar messaging system. With Pulsar Functions, you can write functions of any level of complexity in [Java](#functions-for-java) or [Python](#functions-for-python) and run them in conjunction with a Pulsar cluster without needing to run a separate stream processing engine. -> For a more in-depth overview of the Pulsar Functions feature, see the [Pulsar Functions overview](functions-overview). +> For a more in-depth overview of the Pulsar Functions feature, see the [Pulsar Functions overview](functions-overview.md). ## Core programming model @@ -45,7 +45,7 @@ Some things to note about this Pulsar Function: ### Example deployment -Deploying Pulsar Functions is handled by the [`pulsar-admin`](reference-pulsar-admin) CLI tool, in particular the [`functions`](reference-pulsar-admin.md#functions) command. Here's an example command that would run our [sanitizer](#example-function) function from above in [local run](functions-deploying.md#local-run-mode) mode: +Deploying Pulsar Functions is handled by the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool, in particular the [`functions`](reference-pulsar-admin.md#functions) command. Here's an example command that would run our [sanitizer](#example-function) function from above in [local run](functions-deploying.md#local-run-mode) mode: ```bash @@ -61,7 +61,7 @@ $ bin/pulsar-admin functions localrun \ ``` -For instructions on running functions in your Pulsar cluster, see the [Deploying Pulsar Functions](functions-deploying) guide. +For instructions on running functions in your Pulsar cluster, see the [Deploying Pulsar Functions](functions-deploying.md) guide. ### Available APIs @@ -542,7 +542,7 @@ public class MetricRecorderFunction implements Function { ``` -> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring) guide. +> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring.md) guide. ## Functions for Python @@ -554,7 +554,7 @@ Writing Pulsar Functions in Python entails implementing one of two things: ### Getting started -Regardless of which [deployment mode](functions-deploying) you're using, you'll need to install the following Python libraries on any machine that's running Pulsar Functions written in Python: +Regardless of which [deployment mode](functions-deploying.md) you're using, you'll need to install the following Python libraries on any machine that's running Pulsar Functions written in Python: That could be your local machine for [local run mode](functions-deploying.md#local-run-mode) or a machine running a Pulsar [broker](reference-terminology.md#broker) for [cluster mode](functions-deploying.md#cluster-mode). To install those libraries using pip: diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/functions-deploying.md b/site2/website/versioned_docs/version-2.1.1-incubating/functions-deploying.md index 28276e9deffb3..dfc6dd6795dd2 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/functions-deploying.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/functions-deploying.md @@ -19,8 +19,8 @@ Cluster mode | The function runs *inside of* your Pulsar cluster, on the same ma In order to deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more If you're running a non-[standalone](reference-terminology.md#standalone) cluster, you'll need to obtain the service URL for the cluster. How you obtain the service URL will depend on how you deployed your Pulsar cluster. @@ -52,7 +52,7 @@ Function name | Whichever value is specified for the class name (minus org, libr Tenant | Derived from the input topics' names. If the input topics are under the `marketing` tenant---i.e. the topic names have the form `persistent://marketing/{namespace}/{topicName}`---then the tenant will be `marketing`. Namespace | Derived from the input topics' names. If the input topics are under the `asia` namespace under the `marketing` tenant---i.e. the topic names have the form `persistent://marketing/asia/{topicName}`, then the namespace will be `asia`. Output topic | `{input topic}-{function name}-output`. A function with an input topic name of `incoming` and a function name of `exclamation`, for example, would have an output topic of `incoming-exclamation-output`. -Subscription type | For at-least-once and at-most-once [processing guarantees](functions-guarantees), the [`SHARED`](concepts-messaging.md#shared) is applied by default; for effectively-once guarantees, [`FAILOVER`](concepts-messaging.md#failover) is applied +Subscription type | For at-least-once and at-most-once [processing guarantees](functions-guarantees.md), the [`SHARED`](concepts-messaging.md#shared) is applied by default; for effectively-once guarantees, [`FAILOVER`](concepts-messaging.md#failover) is applied Processing guarantees | [`ATLEAST_ONCE`](functions-guarantees.md) Pulsar service URL | `pulsar://localhost:6650` diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/functions-overview.md b/site2/website/versioned_docs/version-2.1.1-incubating/functions-overview.md index bb2246a764b62..5dbc04204c2b5 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/functions-overview.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/functions-overview.md @@ -154,7 +154,7 @@ class RoutingFunction(Function): ## Command-line interface -Pulsar Functions are managed using the [`pulsar-admin`](reference-pulsar-admin) CLI tool (in particular the [`functions`](reference-pulsar-admin.md#functions) command). Here's an example command that would run a function in [local run mode](#local-run-mode): +Pulsar Functions are managed using the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool (in particular the [`functions`](reference-pulsar-admin.md#functions) command). Here's an example command that would run a function in [local run mode](#local-run-mode): ```bash @@ -492,7 +492,7 @@ $ bin/pulsar-admin functions create \ ## Metrics -Pulsar Functions that use the [Pulsar Functions SDK](#the-pulsar-functions-sdk) can publish metrics to Pulsar. For more information, see [Metrics for Pulsar Functions](functions-metrics). +Pulsar Functions that use the [Pulsar Functions SDK](#the-pulsar-functions-sdk) can publish metrics to Pulsar. For more information, see [Metrics for Pulsar Functions](functions-metrics.md). ## State storage diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/getting-started-docker.md b/site2/website/versioned_docs/version-2.1.1-incubating/getting-started-docker.md index 26908245cb2c9..904d67401668a 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/getting-started-docker.md @@ -43,8 +43,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -103,7 +103,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/getting-started-standalone.md b/site2/website/versioned_docs/version-2.1.1-incubating/getting-started-standalone.md index e814c76d5a339..ed437a88361a3 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For the purposes of local development and testing, you can run Pulsar in standalone mode on your own machine. Standalone mode includes a Pulsar broker as well as the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Run Pulsar Standalone Manually @@ -112,7 +112,7 @@ pulsar-io-twitter-@pulsar:version@.nar > If you are running Pulsar in a bare metal cluster, you need to make sure `connectors` tarball is unzipped in every broker's pulsar directory > (or in every function-worker's pulsar directory if you are running a separate worker cluster for Pulsar functions). > -> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), > you can use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ### Starting the cluster @@ -179,12 +179,12 @@ If the message has been successfully published to the topic, you should see a co ## Using Pulsar clients locally -Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Python](client-libraries-python.md), and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: +Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Python](client-libraries-python.md), and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: * `http://localhost:8080` * `pulsar://localhost:6650` -Here's an example producer for a Pulsar topic using the [Java](client-libraries-java) client: +Here's an example producer for a Pulsar topic using the [Java](client-libraries-java.md) client: ```java @@ -195,7 +195,7 @@ Producer producer = client.newProducer().topic("my-topic").create(); ``` -Here's an example [Python](client-libraries-python) producer: +Here's an example [Python](client-libraries-python.md) producer: ```python @@ -206,7 +206,7 @@ producer = client.create_producer('my-topic') ``` -Finally, here's an example [C++](client-libraries-cpp) producer: +Finally, here's an example [C++](client-libraries-cpp.md) producer: ```cpp diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/io-develop.md b/site2/website/versioned_docs/version-2.1.1-incubating/io-develop.md index 69c9a2f42c428..d361a302b2529 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/io-develop.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/io-develop.md @@ -13,15 +13,15 @@ import TabItem from '@theme/TabItem'; This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -274,7 +274,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -305,7 +305,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). @@ -380,7 +380,7 @@ Pulsar connectors enable you to move data in and out of Pulsar easily. It is imp - Check the metrics provided by Pulsar. - Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. - Set and check your customized metrics. diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/io-managing.md b/site2/website/versioned_docs/version-2.1.1-incubating/io-managing.md index c5c61cc4db7ea..c25e15641efb8 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/io-managing.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/io-managing.md @@ -50,7 +50,7 @@ For details, consult the documentation for [individual connectors](io-overview.m ## Running Connectors -Pulsar connectors can be managed using the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands of the [`pulsar-admin`](reference-pulsar-admin) CLI tool. +Pulsar connectors can be managed using the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands of the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. ### Running sources @@ -166,8 +166,8 @@ Here's an example to submit a Cassandra sink: ## Monitoring Connectors -Since Pulsar IO connectors are running as [Pulsar Functions](functions-overview), so you can use [`functions`](reference-pulsar-admin.md#source) commands -available in the [`pulsar-admin`](reference-pulsar-admin) CLI tool. +Since Pulsar IO connectors are running as [Pulsar Functions](functions-overview.md), so you can use [`functions`](reference-pulsar-admin.md#source) commands +available in the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. ### Retrieve Connector Metadata diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/io-overview.md b/site2/website/versioned_docs/version-2.1.1-incubating/io-overview.md index 2a6ecaee5883f..ea49e4b042e8c 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/io-overview.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/io-overview.md @@ -157,7 +157,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/io-quickstart.md b/site2/website/versioned_docs/version-2.1.1-incubating/io-quickstart.md index 66c9a3c3e5e08..178d70d0a8b63 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/io-quickstart.md @@ -6,14 +6,14 @@ original_id: io-quickstart --- This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O in tandem with running the steps in this guide +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O in tandem with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you will be able to: - Connect your Pulsar cluster with your Cassandra cluster :::tip -1. These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However all +1. These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However all the commands used in this tutorial should be able to be used in a multi-nodes Pulsar cluster without any changes. 2. All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -267,11 +267,11 @@ configs: ``` -To learn more about Cassandra Connector, see [Cassandra Connector](io-cassandra). +To learn more about Cassandra Connector, see [Cassandra Connector](io-cassandra.md). ### Submit a Cassandra Sink -Pulsar provides the [CLI](reference-cli-tools) for running and managing Pulsar I/O connectors. +Pulsar provides the [CLI](reference-cli-tools.md) for running and managing Pulsar I/O connectors. We can run following command to sink a sink connector with type `cassandra` and config file `examples/cassandra-sink.yml`. @@ -296,7 +296,7 @@ as a Pulsar Function and write the messages produced in topic `test_cassandra` t ### Inspect the Cassandra Sink -Since an IO connector is running as [Pulsar Functions](functions-overview), you can use [functions CLI](reference-pulsar-admin.md#functions) +Since an IO connector is running as [Pulsar Functions](functions-overview.md), you can use [functions CLI](reference-pulsar-admin.md#functions) for inspecting and managing the IO connectors. #### Retrieve Sink Info diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/pulsar-2.0.md b/site2/website/versioned_docs/version-2.1.1-incubating/pulsar-2.0.md index 560c8c1bd7cb7..07fba365e9ff2 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/pulsar-2.0.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/pulsar-2.0.md @@ -4,13 +4,13 @@ title: Pulsar 2.0 sidebar_label: "Pulsar 2.0" --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/reference-cli-tools.md b/site2/website/versioned_docs/version-2.1.1-incubating/reference-cli-tools.md index 1c3d601e00c88..dcac11835f4b9 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/reference-cli-tools.md @@ -6,7 +6,7 @@ sidebar_label: "Pulsar CLI tools" Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) @@ -825,7 +825,7 @@ Options ### `transaction` -Run a transaction. For more information, see [Pulsar transactions](txn-why). +Run a transaction. For more information, see [Pulsar transactions](txn-why.md). **Usage** diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/security-athenz.md b/site2/website/versioned_docs/version-2.1.1-incubating/security-athenz.md index ba27ba43eb9d8..6d338fc4013bd 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/security-athenz.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/security-athenz.md @@ -75,7 +75,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/security-authorization.md b/site2/website/versioned_docs/version-2.1.1-incubating/security-authorization.md index d9d33665d066b..c5e3c04b03524 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/security-authorization.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/security-authorization.md @@ -26,7 +26,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -58,7 +58,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -86,7 +86,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication @@ -116,7 +116,7 @@ PulsarAdmin admin = PulsarAdmin.builder() When a client is identified with multiple roles in a token (the type of role claim in the token is an array) during the authentication process, Pulsar supports to check the permissions of all the roles and further authorize the client as long as one of its roles has the required permissions. > **Note**
-> This authorization method is only compatible with [JWT authentication](security-jwt). +> This authorization method is only compatible with [JWT authentication](security-jwt.md). To enable this authorization method, configure the authorization provider as `MultiRolesTokenAuthorizationProvider` in the `conf/broker.conf` file. diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/security-overview.md b/site2/website/versioned_docs/version-2.1.1-incubating/security-overview.md index 2f3fc035ad4cb..3f08cb29ad854 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/security-overview.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/security-overview.md @@ -20,7 +20,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/security-tls-authentication.md b/site2/website/versioned_docs/version-2.1.1-incubating/security-tls-authentication.md index e6921bd44181f..d34ecbc1fdcf3 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/security-tls-authentication.md @@ -6,9 +6,9 @@ sidebar_label: "Authentication using TLS" ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -102,7 +102,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -122,7 +122,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/security-tls-transport.md b/site2/website/versioned_docs/version-2.1.1-incubating/security-tls-transport.md index 87e1960708500..3feaf3a4564ae 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/security-tls-transport.md @@ -8,7 +8,7 @@ sidebar_label: "Transport Encryption using TLS" By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -18,7 +18,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -26,9 +26,9 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). -`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ## Create TLS certificates @@ -146,7 +146,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Configure broker -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -219,7 +219,7 @@ The examples below show that hostname verification is disabled for the CLI tools ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.1.1-incubating/standalone.md b/site2/website/versioned_docs/version-2.1.1-incubating/standalone.md index 6781f9e713003..46a5adebe7464 100644 --- a/site2/website/versioned_docs/version-2.1.1-incubating/standalone.md +++ b/site2/website/versioned_docs/version-2.1.1-incubating/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For the purposes of local development and testing, you can run Pulsar in standalone mode on your own machine. Standalone mode includes a Pulsar broker as well as the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Run Pulsar Standalone Manually @@ -112,7 +112,7 @@ pulsar-io-twitter-@pulsar:version@.nar > If you are running Pulsar in a bare metal cluster, you need to make sure `connectors` tarball is unzipped in every broker's pulsar directory > (or in every function-worker's pulsar directory if you are running a separate worker cluster for Pulsar functions). > -> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), > you can use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ### Starting the cluster @@ -179,12 +179,12 @@ If the message has been successfully published to the topic, you should see a co ## Using Pulsar clients locally -Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Python](client-libraries-python.md), and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: +Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Python](client-libraries-python.md), and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: * `http://localhost:8080` * `pulsar://localhost:6650` -Here's an example producer for a Pulsar topic using the [Java](client-libraries-java) client: +Here's an example producer for a Pulsar topic using the [Java](client-libraries-java.md) client: ```java @@ -195,7 +195,7 @@ Producer producer = client.newProducer().topic("my-topic").create(); ``` -Here's an example [Python](client-libraries-python) producer: +Here's an example [Python](client-libraries-python.md) producer: ```python @@ -206,7 +206,7 @@ producer = client.create_producer('my-topic') ``` -Finally, here's an example [C++](client-libraries-cpp) producer: +Finally, here's an example [C++](client-libraries-cpp.md) producer: ```cpp diff --git a/site2/website/versioned_docs/version-2.10.1-deprecated/admin-api-brokers.md b/site2/website/versioned_docs/version-2.10.1-deprecated/admin-api-brokers.md index 7ac1f7292b824..2674c7da875f9 100644 --- a/site2/website/versioned_docs/version-2.10.1-deprecated/admin-api-brokers.md +++ b/site2/website/versioned_docs/version-2.10.1-deprecated/admin-api-brokers.md @@ -176,7 +176,7 @@ One way to configure a Pulsar [broker](reference-terminology.md#broker) is to su But since all broker configuration in Pulsar is stored in ZooKeeper, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values. -* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. +* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. * In the Pulsar admin {@inject: rest:REST:/} API, dynamic configuration is managed through the `/admin/v2/brokers/configuration` endpoint. ### Update dynamic configuration diff --git a/site2/website/versioned_docs/version-2.10.1-deprecated/schema-manage.md b/site2/website/versioned_docs/version-2.10.1-deprecated/schema-manage.md index b3637473799bd..e62818c7e823f 100644 --- a/site2/website/versioned_docs/version-2.10.1-deprecated/schema-manage.md +++ b/site2/website/versioned_docs/version-2.10.1-deprecated/schema-manage.md @@ -160,7 +160,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.2.0/admin-api-clusters.md b/site2/website/versioned_docs/version-2.2.0/admin-api-clusters.md index 04db2f60759bc..1255ad25d7aac 100644 --- a/site2/website/versioned_docs/version-2.2.0/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.2.0/admin-api-clusters.md @@ -88,7 +88,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -112,11 +112,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block > - For the latest and complete information about `Java admin API`, including classes, methods, descriptions, and more, see [Java admin API doc](https://pulsar.apache.org/api/admin/). -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -121,7 +121,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.2.0/client-libraries-go.md b/site2/website/versioned_docs/version-2.2.0/client-libraries-go.md index d826f69560ff1..6c3b3e9afad52 100644 --- a/site2/website/versioned_docs/version-2.2.0/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.2.0/client-libraries-go.md @@ -16,7 +16,7 @@ The Pulsar Go client can be used to create Pulsar [producers](#producers), [cons ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Installing go package @@ -50,7 +50,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -68,7 +68,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -223,14 +223,14 @@ Parameter | Description | Default :---------|:------------|:------- `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | | `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4) and [`ZLIB`](https://zlib.net/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | ## Consumers @@ -475,7 +475,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.2.0/client-libraries-java.md b/site2/website/versioned_docs/version-2.2.0/client-libraries-java.md index 2c349dbf12f12..1058b0896fa78 100644 --- a/site2/website/versioned_docs/version-2.2.0/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.2.0/client-libraries-java.md @@ -10,7 +10,7 @@ import TabItem from '@theme/TabItem'; ```` -You can use a Pulsar Java client to create the Java [producer](#producer), [consumer](#consumer), [reader](#reader) and [TableView](#tableview) of messages and to perform [administrative tasks](admin-api-overview). The current Java client version is **@pulsar:version@**. +You can use a Pulsar Java client to create the Java [producer](#producer), [consumer](#consumer), [reader](#reader) and [TableView](#tableview) of messages and to perform [administrative tasks](admin-api-overview.md). The current Java client version is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), [reader](#reader) and [TableView](#tableview) of a Java client are thread-safe. @@ -19,10 +19,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | [The producer and consumer API](https://pulsar.apache.org/api/client/) | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Include both `pulsar-client` and `pulsar-client-admin`
    Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.
    In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -69,7 +69,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -95,7 +95,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -126,7 +126,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -279,7 +279,7 @@ Currently, cluster-level failover can perform probes to prevent data loss, but i > #### What are the relationships between cluster-level failover and geo-replication? -The cluster-level failover is an extension of [geo-replication](concepts-replication) to improve stability and robustness. The cluster-level failover depends on geo-replication, and they have some **differences** as below. +The cluster-level failover is an extension of [geo-replication](concepts-replication.md) to improve stability and robustness. The cluster-level failover depends on geo-replication, and they have some **differences** as below. Influence |Cluster-level failover|Geo-replication |---|---|--- @@ -576,7 +576,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics cookbook](cookbooks-partitioned). +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics cookbook](cookbooks-partitioned.md). ### Async send @@ -1370,7 +1370,7 @@ tv.forEach((key, value) -> /*operations on all existing messages*/) ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producer) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producer) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -1488,7 +1488,7 @@ For example of ProtobufNativeSchema, see [`SchemaDefinition` in `Complex type`]( ## Authentication -Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2). You can use the Pulsar Java client with all of them. +Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2.md). You can use the Pulsar Java client with all of them. ### TLS Authentication @@ -1515,7 +1515,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` @@ -1552,7 +1552,7 @@ PulsarClient client = PulsarClient.builder() ### Oauth2 -The following example shows how to use [Oauth2](security-oauth2) as an authentication provider for the Pulsar Java client. +The following example shows how to use [Oauth2](security-oauth2.md) as an authentication provider for the Pulsar Java client. You can use the factory method to configure authentication for Pulsar Java client. diff --git a/site2/website/versioned_docs/version-2.2.0/client-libraries-python.md b/site2/website/versioned_docs/version-2.2.0/client-libraries-python.md index 1d3306c28f687..9de991d4d23a1 100644 --- a/site2/website/versioned_docs/version-2.2.0/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.2.0/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -The Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +The Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. ## Installation diff --git a/site2/website/versioned_docs/version-2.2.0/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.2.0/client-libraries-websocket.md index 839a5308742b2..d5d4457b2a776 100644 --- a/site2/website/versioned_docs/version-2.2.0/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.2.0/client-libraries-websocket.md @@ -5,14 +5,14 @@ sidebar_label: "WebSocket" original_id: client-libraries-websocket --- -Pulsar's [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API is meant to provide a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSockets you can publish and consume messages and use all the features available in the [Java](client-libraries-java.md), [Python](client-libraries-python.md), and [C++](client-libraries-cpp) client libraries. +Pulsar's [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API is meant to provide a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSockets you can publish and consume messages and use all the features available in the [Java](client-libraries-java.md), [Python](client-libraries-python.md), and [C++](client-libraries-cpp.md) client libraries. > You can use Pulsar's WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.2.0/client-libraries.md b/site2/website/versioned_docs/version-2.2.0/client-libraries.md index fbfbfcd113729..a86194d2aabc6 100644 --- a/site2/website/versioned_docs/version-2.2.0/client-libraries.md +++ b/site2/website/versioned_docs/version-2.2.0/client-libraries.md @@ -14,29 +14,29 @@ Pulsar currently has client libraries available for following languages: ## Java client -For a tutorial on using the Pulsar Java client to produce and consume messages, see [The Pulsar Java client](client-libraries-java). +For a tutorial on using the Pulsar Java client to produce and consume messages, see [The Pulsar Java client](client-libraries-java.md). There are also two independent sets of Javadoc API docs available: Library | Purpose :-------|:------- -[`org.apache.pulsar.client.api`](/api/client) | The [Pulsar Java client](client-libraries-java) for producing and consuming messages on Pulsar topics [`org.apache.pulsar.client.admin`](/api/admin) | The Java client for the [Pulsar admin interface](admin-api-overview.md) +[`org.apache.pulsar.client.api`](/api/client) | The [Pulsar Java client](client-libraries-java.md) for producing and consuming messages on Pulsar topics [`org.apache.pulsar.client.admin`](/api/admin) | The Java client for the [Pulsar admin interface](admin-api-overview.md) ## Go client -For a tutorial on using the Pulsar Go client, see [The Pulsar Go client](client-libraries-go). +For a tutorial on using the Pulsar Go client, see [The Pulsar Go client](client-libraries-go.md). ## Python client -For a tutorial on using the Pulsar Python client, see [The Pulsar Python client](client-libraries-python). +For a tutorial on using the Pulsar Python client, see [The Pulsar Python client](client-libraries-python.md). There are also [pdoc](https://github.com/BurntSushi/pdoc)-generated API docs for the Python client [here](/api/python). ## C++ client -For a tutorial on using the Pulsar C++ clent, see [The Pulsar C++ client](client-libraries-cpp). +For a tutorial on using the Pulsar C++ clent, see [The Pulsar C++ client](client-libraries-cpp.md). There are also [Doxygen](http://www.stack.nl/~dimitri/doxygen/)-generated API docs for the C++ client [here](/api/cpp). diff --git a/site2/website/versioned_docs/version-2.2.0/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.2.0/concepts-architecture-overview.md index 0d96b59f8ae91..1ce346fc16941 100644 --- a/site2/website/versioned_docs/version-2.2.0/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.2.0/concepts-architecture-overview.md @@ -4,7 +4,7 @@ title: Architecture Overview sidebar_label: "Architecture" --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -16,20 +16,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers. The producers connect to the brokers to publish messages and the consumers connect to the brokers to consume the messages. -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -39,9 +39,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -139,17 +139,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -157,7 +157,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.2.0/concepts-authentication.md b/site2/website/versioned_docs/version-2.2.0/concepts-authentication.md index 335da8dc5a519..fbefead69f382 100644 --- a/site2/website/versioned_docs/version-2.2.0/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.2.0/concepts-authentication.md @@ -4,5 +4,5 @@ title: Authentication and Authorization sidebar_label: "Authentication and Authorization" --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.2.0/concepts-clients.md b/site2/website/versioned_docs/version-2.2.0/concepts-clients.md index 65201b5ce1cd2..c0fe61cd06e76 100644 --- a/site2/website/versioned_docs/version-2.2.0/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.2.0/concepts-clients.md @@ -4,12 +4,12 @@ title: Pulsar Clients sidebar_label: "Clients" --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -37,7 +37,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.2.0/concepts-messaging.md b/site2/website/versioned_docs/version-2.2.0/concepts-messaging.md index 89b6827d73d5b..11b15cf814aa5 100644 --- a/site2/website/versioned_docs/version-2.2.0/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.2.0/concepts-messaging.md @@ -22,8 +22,8 @@ Messages are the basic "unit" of Pulsar. The following table lists the component Component | Description :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | The key (string type) of the message. It is a short name of message key or partition key. Messages are optionally tagged with keys, which is useful for features like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | The key (string type) of the message. It is a short name of message key or partition key. Messages are optionally tagged with keys, which is useful for features like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer who produces the message. If you do not specify a producer name, the default name is used. Topic name | The name of the topic that the message is published to. @@ -53,7 +53,7 @@ The default size of a message is 5 MB. You can configure the max size of a messa ``` -> For more information on Pulsar messages, see Pulsar [binary protocol](developing-binary-protocol). +> For more information on Pulsar messages, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -171,7 +171,7 @@ Messages are received from [brokers](reference-terminology.md#broker) either syn ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -656,7 +656,7 @@ A [subscription](#subscriptions) can have one or more consumers. When a consumer #### When to use -By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry). +By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry.md). #### How to use @@ -760,7 +760,7 @@ Decisions about routing and subscription modes can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -772,7 +772,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -810,7 +810,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -881,7 +881,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -904,12 +904,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication.md). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.2.0/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.2.0/concepts-multi-tenancy.md index 8a17e72a4c8b3..26bd6a2fd183f 100644 --- a/site2/website/versioned_docs/version-2.2.0/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.2.0/concepts-multi-tenancy.md @@ -4,7 +4,7 @@ title: Multi Tenancy sidebar_label: "Multi Tenancy" --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -20,7 +20,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -28,7 +28,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.2.0/concepts-overview.md b/site2/website/versioned_docs/version-2.2.0/concepts-overview.md index 119d46508ed28..bf4993a3110b5 100644 --- a/site2/website/versioned_docs/version-2.2.0/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.2.0/concepts-overview.md @@ -8,15 +8,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.2.0/concepts-replication.md b/site2/website/versioned_docs/version-2.2.0/concepts-replication.md index d4653ca7aecd8..9cc3a7c3415bd 100644 --- a/site2/website/versioned_docs/version-2.2.0/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.2.0/concepts-replication.md @@ -8,7 +8,7 @@ Regardless of industries, when an unforeseen event occurs and brings day-to-day Pulsar's geo-replication mechanism is typically used for disaster recovery, enabling the replication of persistently stored message data across multiple data centers. For instance, your application is publishing data in one region and you would like to process it for consumption in other regions. With Pulsar’s geo-replication mechanism, messages can be produced and consumed in different geo-locations. -The diagram below illustrates the process of [geo-replication](administration-geo). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. +The diagram below illustrates the process of [geo-replication](administration-geo.md). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. ![A typical geo-replication example with full-mesh pattern](/assets/full-mesh-replication.svg) diff --git a/site2/website/versioned_docs/version-2.2.0/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.2.0/concepts-tiered-storage.md index f0bbde61718fb..ec031253ccd8d 100644 --- a/site2/website/versioned_docs/version-2.2.0/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.2.0/concepts-tiered-storage.md @@ -14,4 +14,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.2.0/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.2.0/concepts-topic-compaction.md index 790ce76b7c5e0..300cf68956dd1 100644 --- a/site2/website/versioned_docs/version-2.2.0/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.2.0/concepts-topic-compaction.md @@ -6,7 +6,7 @@ sidebar_label: "Topic Compaction" Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -23,7 +23,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.2.0/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.2.0/cookbooks-compaction.md index 0720019aaa035..214e899fa086d 100644 --- a/site2/website/versioned_docs/version-2.2.0/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.2.0/cookbooks-compaction.md @@ -44,7 +44,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.2.0/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.2.0/cookbooks-non-persistent.md index d40c4fbd0cf01..1346d19a04263 100644 --- a/site2/website/versioned_docs/version-2.2.0/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.2.0/cookbooks-non-persistent.md @@ -40,7 +40,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -54,7 +54,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.2.0/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.2.0/cookbooks-partitioned.md index 25896934788fc..c857d019edc02 100644 --- a/site2/website/versioned_docs/version-2.2.0/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.2.0/cookbooks-partitioned.md @@ -3,4 +3,4 @@ id: cookbooks-partitioned title: Partitioned topics sidebar_label: "Partitioned Topics" --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.2.0/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.2.0/cookbooks-retention-expiry.md index 86160c3f7ecf8..cd764fad6eb37 100644 --- a/site2/website/versioned_docs/version-2.2.0/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.2.0/cookbooks-retention-expiry.md @@ -21,7 +21,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.2.0/deploy-aws.md b/site2/website/versioned_docs/version-2.2.0/deploy-aws.md index 5c25f945df160..4ab2ea841afa6 100644 --- a/site2/website/versioned_docs/version-2.2.0/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.2.0/deploy-aws.md @@ -5,7 +5,7 @@ sidebar_label: "Amazon Web Services" original_id: deploy-aws --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary to run the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---while Ansible can install and run Pulsar on the provisioned resources. diff --git a/site2/website/versioned_docs/version-2.2.0/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.2.0/deploy-bare-metal-multi-cluster.md index 072c592037c99..42eb52cd467a5 100644 --- a/site2/website/versioned_docs/version-2.2.0/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.2.0/deploy-bare-metal-multi-cluster.md @@ -9,17 +9,17 @@ original_id: deploy-bare-metal-multi-cluster 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you're interested in experimenting with Pulsar or using it in a startup or on a single team, we recommend opting for a single cluster. For instructions on deploying a single cluster, -see the guide [here](deploy-bare-metal). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and make sure it is installed under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and make sure it is installed under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you could reference this [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you could reference this [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar *instance* consists of multiple Pulsar clusters working in unison. Clusters can be distributed across data centers or geographical regions and can replicate amongst themselves using [geo-replication](administration-geo). Deploying a multi-cluster Pulsar instance involves the following basic steps: +A Pulsar *instance* consists of multiple Pulsar clusters working in unison. Clusters can be distributed across data centers or geographical regions and can replicate amongst themselves using [geo-replication](administration-geo.md). Deploying a multi-cluster Pulsar instance involves the following basic steps: * Deploying two separate [ZooKeeper](#deploying-zookeeper) quorums: a [local](#deploying-local-zookeeper) quorum for each cluster in the instance and a [configuration store](#configuration-store) quorum for instance-wide tasks * Initializing [cluster metadata](#cluster-metadata-initialization) for each cluster @@ -29,7 +29,7 @@ A Pulsar *instance* consists of multiple Pulsar clusters working in unison. Clus If you're deploying a single Pulsar cluster, see the [Clusters and Brokers](getting-started-standalone.md#starting-the-cluster) guide. > #### Running Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes. If you'd like to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you're looking to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes. If you'd like to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you're looking to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). ## System requirement Currently, Pulsar is available for 64-bit **macOS**, **Linux**, and **Windows**. To use Pulsar, you need to install 64-bit JRE/JDK 8 or later versions. @@ -246,7 +246,7 @@ As you can see from the example above, the following needs to be specified: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you're using [TLS](security-tls-transport), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -347,17 +347,17 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). -You can also use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can also use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > #### Service discovery already provided by many scheduling systems -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you're running Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you're running Pulsar on such a system, you may not need to provide your own service discovery mechanism. ### Service discovery setup -The service discovery mechanism included with Pulsar maintains a list of active brokers, stored in ZooKeeper, and supports lookup using HTTP and also Pulsar's [binary protocol](developing-binary-protocol). +The service discovery mechanism included with Pulsar maintains a list of active brokers, stored in ZooKeeper, and supports lookup using HTTP and also Pulsar's [binary protocol](developing-binary-protocol.md). To get started setting up Pulsar's built-in service discovery, you need to change a few parameters in the [`conf/discovery.conf`](reference-configuration.md#service-discovery) configuration file. Set the [`zookeeperServers`](reference-configuration.md#service-discovery-zookeeperServers) parameter to the cluster's ZooKeeper quorum connection string and the [`configurationStoreServers`](reference-configuration.md#service-discovery-configurationStoreServers) setting to the [configuration store](reference-terminology.md#configuration-store) quorum connection string. @@ -382,7 +382,7 @@ $ bin/pulsar-daemon start discovery ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.2.0/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.2.0/deploy-bare-metal.md index f6ed859135860..c17d34c1a74db 100644 --- a/site2/website/versioned_docs/version-2.2.0/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.2.0/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you're interested in experimenting with Pulsar or using it in a startup or on a single team, we recommend opting for a single cluster. If you do need to run a multi-cluster Pulsar instance, -however, see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +however, see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and make sure it is installed under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and make sure it is installed under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you could reference this [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you could reference this [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -257,9 +257,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (we don't recommend using a different port). -`--web-service-url-tls` | If you're using [TLS](security-tls-transport), you'll also need to specify a TLS web service URL for the cluster. The default port is 8443 (we don't recommend using a different port). +`--web-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster. The default port is 8443 (we don't recommend using a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (we don't recommend using a different port). -`--broker-service-url-tls` | If you're using [TLS](security-tls-transport), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (we don't recommend using a different port). +`--broker-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (we don't recommend using a different port). ## Deploying a BookKeeper cluster @@ -340,7 +340,7 @@ clusterName=pulsar-cluster-1 ### Enabling Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable function worker, by setting `functionsWorkerEnabled` to `true`. diff --git a/site2/website/versioned_docs/version-2.2.0/deploy-dcos.md b/site2/website/versioned_docs/version-2.2.0/deploy-dcos.md index 4fa966ddc888e..e0fbeae72e4ac 100644 --- a/site2/website/versioned_docs/version-2.2.0/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.2.0/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -If you want to enable all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of +If you want to enable all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.2.0/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.2.0/deploy-kubernetes.md index 4e170dcd4711d..7455247c901d4 100644 --- a/site2/website/versioned_docs/version-2.2.0/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.2.0/deploy-kubernetes.md @@ -5,6 +5,6 @@ sidebar_label: "Kubernetes" --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.2.0/deploy-monitoring.md b/site2/website/versioned_docs/version-2.2.0/deploy-monitoring.md index adf3587b7aeb3..d581fe2edfacb 100644 --- a/site2/website/versioned_docs/version-2.2.0/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.2.0/deploy-monitoring.md @@ -112,7 +112,7 @@ http://$FUNCTIONS_WORKER_ADDRESS:$WORKER_PORT/metrics: You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy). +When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy.md). ## Dashboards @@ -120,7 +120,7 @@ When you collect time series statistics, the major problem is to make sure the n ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager). +The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.2.0/develop-schema.md b/site2/website/versioned_docs/version-2.2.0/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.2.0/develop-schema.md +++ b/site2/website/versioned_docs/version-2.2.0/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.2.0/functions-api.md b/site2/website/versioned_docs/version-2.2.0/functions-api.md index e9f1ffd4407bb..c0c25f9e8a825 100644 --- a/site2/website/versioned_docs/version-2.2.0/functions-api.md +++ b/site2/website/versioned_docs/version-2.2.0/functions-api.md @@ -5,9 +5,9 @@ sidebar_label: "API" original_id: functions-api --- -[Pulsar Functions](functions-overview) provides an easy-to-use API that developers can use to create and manage processing logic for the Apache Pulsar messaging system. With Pulsar Functions, you can write functions of any level of complexity in [Java](#functions-for-java) or [Python](#functions-for-python) and run them in conjunction with a Pulsar cluster without needing to run a separate stream processing engine. +[Pulsar Functions](functions-overview.md) provides an easy-to-use API that developers can use to create and manage processing logic for the Apache Pulsar messaging system. With Pulsar Functions, you can write functions of any level of complexity in [Java](#functions-for-java) or [Python](#functions-for-python) and run them in conjunction with a Pulsar cluster without needing to run a separate stream processing engine. -> For a more in-depth overview of the Pulsar Functions feature, see the [Pulsar Functions overview](functions-overview). +> For a more in-depth overview of the Pulsar Functions feature, see the [Pulsar Functions overview](functions-overview.md). ## Core programming model @@ -45,7 +45,7 @@ Some things to note about this Pulsar Function: ### Example deployment -Deploying Pulsar Functions is handled by the [`pulsar-admin`](reference-pulsar-admin) CLI tool, in particular the [`functions`](reference-pulsar-admin.md#functions) command. Here's an example command that would run our [sanitizer](#example-function) function from above in [local run](functions-deploying.md#local-run-mode) mode: +Deploying Pulsar Functions is handled by the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool, in particular the [`functions`](reference-pulsar-admin.md#functions) command. Here's an example command that would run our [sanitizer](#example-function) function from above in [local run](functions-deploying.md#local-run-mode) mode: ```bash @@ -61,7 +61,7 @@ $ bin/pulsar-admin functions localrun \ ``` -For instructions on running functions in your Pulsar cluster, see the [Deploying Pulsar Functions](functions-deploying) guide. +For instructions on running functions in your Pulsar cluster, see the [Deploying Pulsar Functions](functions-deploying.md) guide. ### Available APIs @@ -101,7 +101,7 @@ Pulsar Functions can take byte arrays as inputs and spit out byte arrays as outp * [SerDe](#SerDe) ### Schema Registry -Pulsar has a built in [Schema Registry](concepts-schema-registry) and comes bundled with a variety of popular schema types(avro, json and protobuf). Pulsar Functions can leverage existing schema information from input topics to derive the input type. The same applies for output topic as well. +Pulsar has a built in [Schema Registry](concepts-schema-registry.md) and comes bundled with a variety of popular schema types(avro, json and protobuf.md). Pulsar Functions can leverage existing schema information from input topics to derive the input type. The same applies for output topic as well. ### SerDe @@ -553,7 +553,7 @@ public class MetricRecorderFunction implements Function { ``` -> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring) guide. +> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring.md) guide. ## Functions for Python @@ -565,7 +565,7 @@ Writing Pulsar Functions in Python entails implementing one of two things: ### Getting started -Regardless of which [deployment mode](functions-deploying) you're using, 'pulsar-client' python library has to installed on any machine that's running Pulsar Functions written in Python. +Regardless of which [deployment mode](functions-deploying.md) you're using, 'pulsar-client' python library has to installed on any machine that's running Pulsar Functions written in Python. That could be your local machine for [local run mode](functions-deploying.md#local-run-mode) or a machine running a Pulsar [broker](reference-terminology.md#broker) for [cluster mode](functions-deploying.md#cluster-mode). To install those libraries using pip: diff --git a/site2/website/versioned_docs/version-2.2.0/functions-deploying.md b/site2/website/versioned_docs/version-2.2.0/functions-deploying.md index 19565c1c4389c..d5395324ef864 100644 --- a/site2/website/versioned_docs/version-2.2.0/functions-deploying.md +++ b/site2/website/versioned_docs/version-2.2.0/functions-deploying.md @@ -19,8 +19,8 @@ Cluster mode | The function runs *inside of* your Pulsar cluster, on the same ma In order to deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more If you're running a non-[standalone](reference-terminology.md#standalone) cluster, you'll need to obtain the service URL for the cluster. How you obtain the service URL will depend on how you deployed your Pulsar cluster. @@ -52,7 +52,7 @@ Function name | Whichever value is specified for the class name (minus org, libr Tenant | Derived from the input topics' names. If the input topics are under the `marketing` tenant---i.e. the topic names have the form `persistent://marketing/{namespace}/{topicName}`---then the tenant will be `marketing`. Namespace | Derived from the input topics' names. If the input topics are under the `asia` namespace under the `marketing` tenant---i.e. the topic names have the form `persistent://marketing/asia/{topicName}`, then the namespace will be `asia`. Output topic | `{input topic}-{function name}-output`. A function with an input topic name of `incoming` and a function name of `exclamation`, for example, would have an output topic of `incoming-exclamation-output`. -Subscription type | For at-least-once and at-most-once [processing guarantees](functions-guarantees), the [`SHARED`](concepts-messaging.md#shared) is applied by default; for effectively-once guarantees, [`FAILOVER`](concepts-messaging.md#failover) is applied +Subscription type | For at-least-once and at-most-once [processing guarantees](functions-guarantees.md), the [`SHARED`](concepts-messaging.md#shared) is applied by default; for effectively-once guarantees, [`FAILOVER`](concepts-messaging.md#failover) is applied Processing guarantees | [`ATLEAST_ONCE`](functions-guarantees.md) Pulsar service URL | `pulsar://localhost:6650` diff --git a/site2/website/versioned_docs/version-2.2.0/functions-overview.md b/site2/website/versioned_docs/version-2.2.0/functions-overview.md index 3bdfbaf97b830..4c6251e088f1a 100644 --- a/site2/website/versioned_docs/version-2.2.0/functions-overview.md +++ b/site2/website/versioned_docs/version-2.2.0/functions-overview.md @@ -154,7 +154,7 @@ class RoutingFunction(Function): ## Command-line interface -Pulsar Functions are managed using the [`pulsar-admin`](reference-pulsar-admin) CLI tool (in particular the [`functions`](reference-pulsar-admin.md#functions) command). Here's an example command that would run a function in [local run mode](#local-run-mode): +Pulsar Functions are managed using the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool (in particular the [`functions`](reference-pulsar-admin.md#functions) command). Here's an example command that would run a function in [local run mode](#local-run-mode): ```bash @@ -492,7 +492,7 @@ $ bin/pulsar-admin functions create \ ## Metrics -Pulsar Functions that use the [Pulsar Functions SDK](#the-pulsar-functions-sdk) can publish metrics to Pulsar. For more information, see [Metrics for Pulsar Functions](functions-metrics). +Pulsar Functions that use the [Pulsar Functions SDK](#the-pulsar-functions-sdk) can publish metrics to Pulsar. For more information, see [Metrics for Pulsar Functions](functions-metrics.md). ## State storage diff --git a/site2/website/versioned_docs/version-2.2.0/getting-started-docker.md b/site2/website/versioned_docs/version-2.2.0/getting-started-docker.md index 26908245cb2c9..904d67401668a 100644 --- a/site2/website/versioned_docs/version-2.2.0/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.2.0/getting-started-docker.md @@ -43,8 +43,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -103,7 +103,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.2.0/getting-started-standalone.md b/site2/website/versioned_docs/version-2.2.0/getting-started-standalone.md index 59a19174b2ea8..8984d7a9b9c4e 100644 --- a/site2/website/versioned_docs/version-2.2.0/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.2.0/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For the purposes of local development and testing, you can run Pulsar in standalone mode on your own machine. Standalone mode includes a Pulsar broker as well as the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Run Pulsar Standalone Manually @@ -112,7 +112,7 @@ pulsar-io-twitter-@pulsar:version@.nar > If you are running Pulsar in a bare metal cluster, you need to make sure `connectors` tarball is unzipped in every broker's pulsar directory > (or in every function-worker's pulsar directory if you are running a separate worker cluster for Pulsar functions). > -> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), > you can use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ## Installing Tiered Storage Offloaders (optional) @@ -121,7 +121,7 @@ pulsar-io-twitter-@pulsar:version@.nar > If you would like to enable tiered storage feature, you can follow the instructions as below; otherwise you can > skip this section for now. -To get started using [tiered storage offloaders](concepts-tiered-storage), you'll need to download the offloaders tarball release on every broker node in +To get started using [tiered storage offloaders](concepts-tiered-storage.md), you'll need to download the offloaders tarball release on every broker node in one of the following ways: * by clicking the link below and downloading the release from an Apache mirror: @@ -161,7 +161,7 @@ For more details of how to configure tiered storage feature, you could reference > > If you are running Pulsar in a bare metal cluster, you need to make sure `offloaders` tarball is unzipped in every broker's pulsar directory > -> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), > you can use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. @@ -229,12 +229,12 @@ If the message has been successfully published to the topic, you should see a co ## Using Pulsar clients locally -Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Python](client-libraries-python.md), and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: +Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Python](client-libraries-python.md), and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: * `http://localhost:8080` * `pulsar://localhost:6650` -Here's an example producer for a Pulsar topic using the [Java](client-libraries-java) client: +Here's an example producer for a Pulsar topic using the [Java](client-libraries-java.md) client: ```java @@ -245,7 +245,7 @@ Producer producer = client.newProducer().topic("my-topic").create(); ``` -Here's an example [Python](client-libraries-python) producer: +Here's an example [Python](client-libraries-python.md) producer: ```python @@ -256,7 +256,7 @@ producer = client.create_producer('my-topic') ``` -Finally, here's an example [C++](client-libraries-cpp) producer: +Finally, here's an example [C++](client-libraries-cpp.md) producer: ```cpp diff --git a/site2/website/versioned_docs/version-2.2.0/io-develop.md b/site2/website/versioned_docs/version-2.2.0/io-develop.md index 69c9a2f42c428..d361a302b2529 100644 --- a/site2/website/versioned_docs/version-2.2.0/io-develop.md +++ b/site2/website/versioned_docs/version-2.2.0/io-develop.md @@ -13,15 +13,15 @@ import TabItem from '@theme/TabItem'; This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -274,7 +274,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -305,7 +305,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). @@ -380,7 +380,7 @@ Pulsar connectors enable you to move data in and out of Pulsar easily. It is imp - Check the metrics provided by Pulsar. - Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. - Set and check your customized metrics. diff --git a/site2/website/versioned_docs/version-2.2.0/io-managing.md b/site2/website/versioned_docs/version-2.2.0/io-managing.md index 47a4922b38dbd..c2e49d8f05f1a 100644 --- a/site2/website/versioned_docs/version-2.2.0/io-managing.md +++ b/site2/website/versioned_docs/version-2.2.0/io-managing.md @@ -50,7 +50,7 @@ For details, consult the documentation for [individual connectors](io-overview.m ## Running Connectors -Pulsar connectors can be managed using the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands of the [`pulsar-admin`](reference-pulsar-admin) CLI tool. +Pulsar connectors can be managed using the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands of the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. ### Running sources @@ -166,8 +166,8 @@ Here's an example to submit a Cassandra sink: ## Monitoring Connectors -Since Pulsar IO connectors are running as [Pulsar Functions](functions-overview), so you can use [`functions`](reference-pulsar-admin.md#source) commands -available in the [`pulsar-admin`](reference-pulsar-admin) CLI tool. +Since Pulsar IO connectors are running as [Pulsar Functions](functions-overview.md), so you can use [`functions`](reference-pulsar-admin.md#source) commands +available in the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. ### Retrieve Connector Metadata diff --git a/site2/website/versioned_docs/version-2.2.0/io-overview.md b/site2/website/versioned_docs/version-2.2.0/io-overview.md index 4b238dac32f1f..e1c093dc3a968 100644 --- a/site2/website/versioned_docs/version-2.2.0/io-overview.md +++ b/site2/website/versioned_docs/version-2.2.0/io-overview.md @@ -8,7 +8,7 @@ original_id: io-overview Messaging systems are most powerful when you can easily use them in conjunction with external systems like databases and other messaging systems. **Pulsar IO** is a feature of Pulsar that enables you to easily create, deploy, and manage Pulsar **connectors** that interact with external systems, such as [Apache Cassandra](https://cassandra.apache.org), [Aerospike](https://www.aerospike.com), and many others. > #### Pulsar IO and Pulsar Functions -> Under the hood, Pulsar IO connectors are specialized [Pulsar Functions](functions-overview.md) purpose-built to interface with external systems. The [administrative interface](io-quickstart) for Pulsar IO is, in fact, quite similar to that of Pulsar Functions. +> Under the hood, Pulsar IO connectors are specialized [Pulsar Functions](functions-overview.md) purpose-built to interface with external systems. The [administrative interface](io-quickstart.md) for Pulsar IO is, in fact, quite similar to that of Pulsar Functions. ## Sources and sinks @@ -23,7 +23,7 @@ This diagram illustrates the relationship between sources, sinks, and Pulsar: ## Working with connectors -Pulsar IO connectors can be managed via the [`pulsar-admin`](reference-pulsar-admin) CLI tool, in particular the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands. +Pulsar IO connectors can be managed via the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool, in particular the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands. > For a guide to managing connectors in your Pulsar installation, see the [Getting started with Pulsar IO](io-quickstart.md) @@ -31,8 +31,8 @@ The following connectors are currently available for Pulsar: |Name|Java Class|Documentation| |---|---|---| -|[Aerospike sink](https://www.aerospike.com/)|[`org.apache.pulsar.io.aerospike.AerospikeSink`](https://github.com/apache/pulsar/blob/master/pulsar-io/aerospike/src/main/java/org/apache/pulsar/io/aerospike/AerospikeStringSink.java)|[Documentation](io-aerospike)| -|[Cassandra sink](https://cassandra.apache.org)|[`org.apache.pulsar.io.cassandra.CassandraSink`](https://github.com/apache/pulsar/blob/master/pulsar-io/cassandra/src/main/java/org/apache/pulsar/io/cassandra/CassandraStringSink.java)|[Documentation](io-cassandra)| +|[Aerospike sink](https://www.aerospike.com/)|[`org.apache.pulsar.io.aerospike.AerospikeSink`](https://github.com/apache/pulsar/blob/master/pulsar-io/aerospike/src/main/java/org/apache/pulsar/io/aerospike/AerospikeStringSink.java)|[Documentation](io-aerospike.md)| +|[Cassandra sink](https://cassandra.apache.org)|[`org.apache.pulsar.io.cassandra.CassandraSink`](https://github.com/apache/pulsar/blob/master/pulsar-io/cassandra/src/main/java/org/apache/pulsar/io/cassandra/CassandraStringSink.java)|[Documentation](io-cassandra.md)| |[Kafka source](https://kafka.apache.org)|[`org.apache.pulsar.io.kafka.KafkaSource`](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaStringSource.java)|[Documentation](io-kafka.md#source)| |[Kafka sink](https://kafka.apache.org)|[`org.apache.pulsar.io.kafka.KafkaSink`](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaStringSink.java)|[Documentation](io-kafka.md#sink)| |[Kinesis sink](https://aws.amazon.com/kinesis/)|[`org.apache.pulsar.io.kinesis.KinesisSink`](https://github.com/apache/pulsar/blob/master/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java)|[Documentation](io-kinesis.md#sink)| diff --git a/site2/website/versioned_docs/version-2.2.0/io-quickstart.md b/site2/website/versioned_docs/version-2.2.0/io-quickstart.md index a6541d8e928ac..404c94979de21 100644 --- a/site2/website/versioned_docs/version-2.2.0/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.2.0/io-quickstart.md @@ -6,14 +6,14 @@ original_id: io-quickstart --- This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O in tandem with running the steps in this guide +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O in tandem with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you will be able to: - Connect your Pulsar cluster with your Cassandra cluster :::tip -1. These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However all +1. These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However all the commands used in this tutorial should be able to be used in a multi-nodes Pulsar cluster without any changes. 2. All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -267,11 +267,11 @@ configs: ``` -To learn more about Cassandra Connector, see [Cassandra Connector](io-cassandra). +To learn more about Cassandra Connector, see [Cassandra Connector](io-cassandra.md). ### Submit a Cassandra Sink -Pulsar provides the [CLI](reference-cli-tools) for running and managing Pulsar I/O connectors. +Pulsar provides the [CLI](reference-cli-tools.md) for running and managing Pulsar I/O connectors. We can run following command to sink a sink connector with type `cassandra` and config file `examples/cassandra-sink.yml`. @@ -296,7 +296,7 @@ as a Pulsar Function and write the messages produced in topic `test_cassandra` t ### Inspect the Cassandra Sink -Since an IO connector is running as [Pulsar Functions](functions-overview), you can use [functions CLI](reference-pulsar-admin.md#functions) +Since an IO connector is running as [Pulsar Functions](functions-overview.md), you can use [functions CLI](reference-pulsar-admin.md#functions) for inspecting and managing the IO connectors. #### Retrieve Sink Info diff --git a/site2/website/versioned_docs/version-2.2.0/pulsar-2.0.md b/site2/website/versioned_docs/version-2.2.0/pulsar-2.0.md index 560c8c1bd7cb7..07fba365e9ff2 100644 --- a/site2/website/versioned_docs/version-2.2.0/pulsar-2.0.md +++ b/site2/website/versioned_docs/version-2.2.0/pulsar-2.0.md @@ -4,13 +4,13 @@ title: Pulsar 2.0 sidebar_label: "Pulsar 2.0" --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.2.0/reference-cli-tools.md b/site2/website/versioned_docs/version-2.2.0/reference-cli-tools.md index 1c3d601e00c88..dcac11835f4b9 100644 --- a/site2/website/versioned_docs/version-2.2.0/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.2.0/reference-cli-tools.md @@ -6,7 +6,7 @@ sidebar_label: "Pulsar CLI tools" Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) @@ -825,7 +825,7 @@ Options ### `transaction` -Run a transaction. For more information, see [Pulsar transactions](txn-why). +Run a transaction. For more information, see [Pulsar transactions](txn-why.md). **Usage** diff --git a/site2/website/versioned_docs/version-2.2.0/security-athenz.md b/site2/website/versioned_docs/version-2.2.0/security-athenz.md index ba27ba43eb9d8..6d338fc4013bd 100644 --- a/site2/website/versioned_docs/version-2.2.0/security-athenz.md +++ b/site2/website/versioned_docs/version-2.2.0/security-athenz.md @@ -75,7 +75,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.2.0/security-authorization.md b/site2/website/versioned_docs/version-2.2.0/security-authorization.md index d9d33665d066b..c5e3c04b03524 100644 --- a/site2/website/versioned_docs/version-2.2.0/security-authorization.md +++ b/site2/website/versioned_docs/version-2.2.0/security-authorization.md @@ -26,7 +26,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -58,7 +58,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -86,7 +86,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication @@ -116,7 +116,7 @@ PulsarAdmin admin = PulsarAdmin.builder() When a client is identified with multiple roles in a token (the type of role claim in the token is an array) during the authentication process, Pulsar supports to check the permissions of all the roles and further authorize the client as long as one of its roles has the required permissions. > **Note**
    -> This authorization method is only compatible with [JWT authentication](security-jwt). +> This authorization method is only compatible with [JWT authentication](security-jwt.md). To enable this authorization method, configure the authorization provider as `MultiRolesTokenAuthorizationProvider` in the `conf/broker.conf` file. diff --git a/site2/website/versioned_docs/version-2.2.0/security-extending.md b/site2/website/versioned_docs/version-2.2.0/security-extending.md index 97edb6898db1e..deba98da936ac 100644 --- a/site2/website/versioned_docs/version-2.2.0/security-extending.md +++ b/site2/website/versioned_docs/version-2.2.0/security-extending.md @@ -10,7 +10,7 @@ Pulsar provides a way to use custom authentication and authorization mechanisms ## Authentication Pulsar support mutual TLS and Athenz authentication plugins, and these can be used as described -in [Security](security-overview). +in [Security](security-overview.md). It is possible to use a custom authentication mechanism by providing the implementation in the form of two plugins one for the Client library and the other for the Pulsar Broker to validate diff --git a/site2/website/versioned_docs/version-2.2.0/security-overview.md b/site2/website/versioned_docs/version-2.2.0/security-overview.md index 2f3fc035ad4cb..3f08cb29ad854 100644 --- a/site2/website/versioned_docs/version-2.2.0/security-overview.md +++ b/site2/website/versioned_docs/version-2.2.0/security-overview.md @@ -20,7 +20,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.2.0/security-tls-authentication.md b/site2/website/versioned_docs/version-2.2.0/security-tls-authentication.md index e6921bd44181f..d34ecbc1fdcf3 100644 --- a/site2/website/versioned_docs/version-2.2.0/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.2.0/security-tls-authentication.md @@ -6,9 +6,9 @@ sidebar_label: "Authentication using TLS" ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -102,7 +102,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -122,7 +122,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.2.0/security-tls-transport.md b/site2/website/versioned_docs/version-2.2.0/security-tls-transport.md index 7f7d1d4749cc7..6cfe4df2e0ed0 100644 --- a/site2/website/versioned_docs/version-2.2.0/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.2.0/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text, which means that all data is sent in the clear. TLS can be used to encrypt this traffic so that it cannot be snooped by a man-in-the-middle attacker. -TLS can be configured for both encryption and authentication. You may configure just TLS transport encryption, which is covered in this guide. TLS authentication is covered [elsewhere](security-tls-authentication.md). Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +TLS can be configured for both encryption and authentication. You may configure just TLS transport encryption, which is covered in this guide. TLS authentication is covered [elsewhere](security-tls-authentication.md). Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may have a performance impact due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -A third kind of key pair, **client key pairs**, are used for [client authentication](security-tls-authentication). +A third kind of key pair, **client key pairs**, are used for [client authentication](security-tls-authentication.md). The **certificate authority** private key should be stored in a very secure location (a fully encrypted, disconnected, air gapped computer). The certificate authority public key, the **trust cert**, can be freely shared. @@ -27,7 +27,7 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server they are talking to has a key pair that was signed by the certificate authority. A man-in-the-middle attacker would not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that was signed by the certificate authority. The Common Name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that was signed by the certificate authority. The Common Name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). ## Creating TLS Certificates @@ -128,7 +128,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Broker Configuration -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you'll need to make some changes to `broker.conf`, which is located in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you'll need to make some changes to `broker.conf`, which is located in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -180,7 +180,7 @@ The examples below show hostname verification being disabled for the Java client ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You'll need to add the following parameters to that file to use TLS transport with Pulsar's CLI tools: diff --git a/site2/website/versioned_docs/version-2.2.0/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.2.0/sql-deployment-configurations.md index 2fa91159ae6be..9a9e91e29efbe 100644 --- a/site2/website/versioned_docs/version-2.2.0/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.2.0/sql-deployment-configurations.md @@ -42,7 +42,7 @@ $ wget pulsar:binary_release_url ## Deploying a new cluster -Please note that the [Getting Started](sql-getting-started) guide shows you how to easily setup a standalone single node environment to experiment with. +Please note that the [Getting Started](sql-getting-started.md) guide shows you how to easily setup a standalone single node environment to experiment with. Pulsar SQL is powered by [Presto](https://prestosql.io) thus many of the configurations for deployment is the same for the Pulsar SQL worker. diff --git a/site2/website/versioned_docs/version-2.2.0/sql-overview.md b/site2/website/versioned_docs/version-2.2.0/sql-overview.md index 025fcd9a808ea..f5748a1b864a4 100644 --- a/site2/website/versioned_docs/version-2.2.0/sql-overview.md +++ b/site2/website/versioned_docs/version-2.2.0/sql-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Overview" original_id: sql-overview --- -One of the common use cases of Pulsar is storing streams of event data. Often the event data is structured which predefined fields. There is tremendous value for users to be able to query the existing data that is already stored in Pulsar topics. With the implementation of the [Schema Registry](concepts-schema-registry), structured data can be stored in Pulsar and allows for the potential to query that data via SQL language. +One of the common use cases of Pulsar is storing streams of event data. Often the event data is structured which predefined fields. There is tremendous value for users to be able to query the existing data that is already stored in Pulsar topics. With the implementation of the [Schema Registry](concepts-schema-registry.md), structured data can be stored in Pulsar and allows for the potential to query that data via SQL language. By leveraging [Presto](https://prestosql.io/), we have created a method for users to be able to query structured data stored within Pulsar in a very efficient and scalable manner. We will discuss why this very efficient and scalable in the [Performance](#performance) section below. diff --git a/site2/website/versioned_docs/version-2.2.0/standalone-docker.md b/site2/website/versioned_docs/version-2.2.0/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.2.0/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.2.0/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.2.0/standalone.md b/site2/website/versioned_docs/version-2.2.0/standalone.md index 7297d79235327..0d595d44aed58 100644 --- a/site2/website/versioned_docs/version-2.2.0/standalone.md +++ b/site2/website/versioned_docs/version-2.2.0/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For the purposes of local development and testing, you can run Pulsar in standalone mode on your own machine. Standalone mode includes a Pulsar broker as well as the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Run Pulsar Standalone Manually @@ -112,7 +112,7 @@ pulsar-io-twitter-@pulsar:version@.nar > If you are running Pulsar in a bare metal cluster, you need to make sure `connectors` tarball is unzipped in every broker's pulsar directory > (or in every function-worker's pulsar directory if you are running a separate worker cluster for Pulsar functions). > -> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), > you can use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ## Installing Tiered Storage Offloaders (optional) @@ -121,7 +121,7 @@ pulsar-io-twitter-@pulsar:version@.nar > If you would like to enable tiered storage feature, you can follow the instructions as below; otherwise you can > skip this section for now. -To get started using [tiered storage offloaders](concepts-tiered-storage), you'll need to download the offloaders tarball release on every broker node in +To get started using [tiered storage offloaders](concepts-tiered-storage.md), you'll need to download the offloaders tarball release on every broker node in one of the following ways: * by clicking the link below and downloading the release from an Apache mirror: @@ -161,7 +161,7 @@ For more details of how to configure tiered storage feature, you could reference > > If you are running Pulsar in a bare metal cluster, you need to make sure `offloaders` tarball is unzipped in every broker's pulsar directory > -> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), > you can use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. @@ -229,12 +229,12 @@ If the message has been successfully published to the topic, you should see a co ## Using Pulsar clients locally -Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Python](client-libraries-python.md), and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: +Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Python](client-libraries-python.md), and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: * `http://localhost:8080` * `pulsar://localhost:6650` -Here's an example producer for a Pulsar topic using the [Java](client-libraries-java) client: +Here's an example producer for a Pulsar topic using the [Java](client-libraries-java.md) client: ```java @@ -245,7 +245,7 @@ Producer producer = client.newProducer().topic("my-topic").create(); ``` -Here's an example [Python](client-libraries-python) producer: +Here's an example [Python](client-libraries-python.md) producer: ```python @@ -256,7 +256,7 @@ producer = client.create_producer('my-topic') ``` -Finally, here's an example [C++](client-libraries-cpp) producer: +Finally, here's an example [C++](client-libraries-cpp.md) producer: ```cpp diff --git a/site2/website/versioned_docs/version-2.2.1/admin-api-clusters.md b/site2/website/versioned_docs/version-2.2.1/admin-api-clusters.md index 04db2f60759bc..1255ad25d7aac 100644 --- a/site2/website/versioned_docs/version-2.2.1/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.2.1/admin-api-clusters.md @@ -88,7 +88,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -112,11 +112,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block > - For the latest and complete information about `Java admin API`, including classes, methods, descriptions, and more, see [Java admin API doc](https://pulsar.apache.org/api/admin/). -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -121,7 +121,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.2.1/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.2.1/client-libraries-cpp.md index 5ba9e03d89521..b709463ff0309 100644 --- a/site2/website/versioned_docs/version-2.2.1/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.2.1/client-libraries-cpp.md @@ -661,7 +661,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema This section describes some examples about schema. For more information about -schema, see [Pulsar schema](schema-get-started). +schema, see [Pulsar schema](schema-get-started.md). ### Avro schema diff --git a/site2/website/versioned_docs/version-2.2.1/client-libraries-go.md b/site2/website/versioned_docs/version-2.2.1/client-libraries-go.md index a1ab520b66192..2cc2f4c901337 100644 --- a/site2/website/versioned_docs/version-2.2.1/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.2.1/client-libraries-go.md @@ -16,7 +16,7 @@ The Pulsar Go client can be used to create Pulsar [producers](#producers), [cons ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Installing go package @@ -50,7 +50,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -68,7 +68,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -223,14 +223,14 @@ Parameter | Description | Default :---------|:------------|:------- `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | | `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4) and [`ZLIB`](https://zlib.net/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | ## Consumers @@ -475,7 +475,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.2.1/client-libraries-java.md b/site2/website/versioned_docs/version-2.2.1/client-libraries-java.md index c33b1a68be579..a6a930249ed86 100644 --- a/site2/website/versioned_docs/version-2.2.1/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.2.1/client-libraries-java.md @@ -5,16 +5,16 @@ sidebar_label: "Java" original_id: client-libraries-java --- -The Pulsar Java client can be used both to create Java producers, consumers, and [readers](#readers) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +The Pulsar Java client can be used both to create Java producers, consumers, and [readers](#readers) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. Javadoc for the Pulsar client is divided up into two domains, by package: Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) -This document will focus only on the client API for producing and consuming messages on Pulsar topics. For a guide to using the Java admin client, see [The Pulsar admin interface](admin-api-overview). +This document will focus only on the client API for producing and consuming messages on Pulsar topics. For a guide to using the Java admin client, see [The Pulsar admin interface](admin-api-overview.md). ## Installation @@ -54,7 +54,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -72,7 +72,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -93,7 +93,7 @@ PulsarClient client = PulsarClient.builder() ``` > #### Default broker URLs for standalone clusters -> If you're running a cluster in [standalone mode](getting-started-standalone), the broker will be available at the `pulsar://localhost:6650` URL by default. +> If you're running a cluster in [standalone mode](getting-started-standalone.md), the broker will be available at the `pulsar://localhost:6650` URL by default. Check out the Javadoc for the {@inject: javadoc:PulsarClient:/client/org/apache/pulsar/client/api/PulsarClient} class for a full listing of configurable parameters. @@ -170,7 +170,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -375,7 +375,7 @@ The code sample above shows pointing the `Reader` object to a specific message ( ## Schemas -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](concepts-schema-registry) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. Here's an example: +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](concepts-schema-registry.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. Here's an example: ```java @@ -470,7 +470,7 @@ The following schema formats are currently available for Java: ## Authentication -Pulsar currently supports two authentication schemes: [TLS](security-tls-authentication.md) and [Athenz](security-athenz). The Pulsar Java client can be used with both. +Pulsar currently supports two authentication schemes: [TLS](security-tls-authentication.md) and [Athenz](security-athenz.md). The Pulsar Java client can be used with both. ### TLS Authentication @@ -497,7 +497,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` diff --git a/site2/website/versioned_docs/version-2.2.1/client-libraries-python.md b/site2/website/versioned_docs/version-2.2.1/client-libraries-python.md index 4b6b4f9e41d6c..7ed649e86096a 100644 --- a/site2/website/versioned_docs/version-2.2.1/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.2.1/client-libraries-python.md @@ -10,7 +10,7 @@ import TabItem from '@theme/TabItem'; ```` -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [Python directory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [Python directory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. @@ -192,7 +192,7 @@ client.close() ``` ### Create a Python client with multiple advertised listeners -To ensure clients in both internal and external networks can connect to a Pulsar cluster, Pulsar introduces [advertisedListeners](concepts-multiple-advertised-listeners). +To ensure clients in both internal and external networks can connect to a Pulsar cluster, Pulsar introduces [advertisedListeners](concepts-multiple-advertised-listeners.md). The following example creates a Python client using multiple advertised listeners: diff --git a/site2/website/versioned_docs/version-2.2.1/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.2.1/client-libraries-websocket.md index a7cca18b0fe57..5c8d6444d5c27 100644 --- a/site2/website/versioned_docs/version-2.2.1/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.2.1/client-libraries-websocket.md @@ -5,14 +5,14 @@ sidebar_label: "WebSocket" original_id: client-libraries-websocket --- -Pulsar's [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API is meant to provide a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSockets you can publish and consume messages and use all the features available in the [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp) client libraries. +Pulsar's [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API is meant to provide a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSockets you can publish and consume messages and use all the features available in the [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md) client libraries. > You can use Pulsar's WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.2.1/client-libraries.md b/site2/website/versioned_docs/version-2.2.1/client-libraries.md index 536cd0ccda82f..8f91dfbbbdd8c 100644 --- a/site2/website/versioned_docs/version-2.2.1/client-libraries.md +++ b/site2/website/versioned_docs/version-2.2.1/client-libraries.md @@ -8,12 +8,12 @@ Pulsar supports the following client libraries: |Language|Documentation|Release note|Code repo |---|---|---|--- -Java |- [User doc](client-libraries-java)

    - [API doc](https://pulsar.apache.org/api/client/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client) -C++ | - [User doc](client-libraries-cpp)

    - [API doc](https://pulsar.apache.org/api/cpp/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp) -Python | - [User doc](client-libraries-python)

    - [API doc](https://pulsar.apache.org/api/python/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) -WebSocket| [User doc](client-libraries-websocket) | [Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-websocket) +Java |- [User doc](client-libraries-java.md)

    - [API doc](https://pulsar.apache.org/api/client/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client) +C++ | - [User doc](client-libraries-cpp.md)

    - [API doc](https://pulsar.apache.org/api/cpp/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp) +Python | - [User doc](client-libraries-python.md)

    - [API doc](https://pulsar.apache.org/api/python/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) +WebSocket| [User doc](client-libraries-websocket.md) | [Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-websocket) Go client|[User doc](client-libraries-go.md)|[Here](https://github.com/apache/pulsar-client-go/blob/master/CHANGELOG) |[Here](https://github.com/apache/pulsar-client-go) -Node.js|[User doc](client-libraries-node)|[Here](https://github.com/apache/pulsar-client-node/releases) |[Here](https://github.com/apache/pulsar-client-node) +Node.js|[User doc](client-libraries-node.md)|[Here](https://github.com/apache/pulsar-client-node/releases) |[Here](https://github.com/apache/pulsar-client-node) C# |[User doc](client-libraries-dotnet.md)| [Here](https://github.com/apache/pulsar-dotpulsar/blob/master/CHANGELOG)|[Here](https://github.com/apache/pulsar-dotpulsar) :::note diff --git a/site2/website/versioned_docs/version-2.2.1/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.2.1/concepts-architecture-overview.md index 0d96b59f8ae91..1ce346fc16941 100644 --- a/site2/website/versioned_docs/version-2.2.1/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.2.1/concepts-architecture-overview.md @@ -4,7 +4,7 @@ title: Architecture Overview sidebar_label: "Architecture" --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -16,20 +16,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers. The producers connect to the brokers to publish messages and the consumers connect to the brokers to consume the messages. -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -39,9 +39,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -139,17 +139,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -157,7 +157,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.2.1/concepts-authentication.md b/site2/website/versioned_docs/version-2.2.1/concepts-authentication.md index 335da8dc5a519..fbefead69f382 100644 --- a/site2/website/versioned_docs/version-2.2.1/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.2.1/concepts-authentication.md @@ -4,5 +4,5 @@ title: Authentication and Authorization sidebar_label: "Authentication and Authorization" --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.2.1/concepts-clients.md b/site2/website/versioned_docs/version-2.2.1/concepts-clients.md index b9a3940f2ad3b..a780c3ff33e37 100644 --- a/site2/website/versioned_docs/version-2.2.1/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.2.1/concepts-clients.md @@ -5,7 +5,7 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. diff --git a/site2/website/versioned_docs/version-2.2.1/concepts-messaging.md b/site2/website/versioned_docs/version-2.2.1/concepts-messaging.md index 89b6827d73d5b..11b15cf814aa5 100644 --- a/site2/website/versioned_docs/version-2.2.1/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.2.1/concepts-messaging.md @@ -22,8 +22,8 @@ Messages are the basic "unit" of Pulsar. The following table lists the component Component | Description :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | The key (string type) of the message. It is a short name of message key or partition key. Messages are optionally tagged with keys, which is useful for features like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | The key (string type) of the message. It is a short name of message key or partition key. Messages are optionally tagged with keys, which is useful for features like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer who produces the message. If you do not specify a producer name, the default name is used. Topic name | The name of the topic that the message is published to. @@ -53,7 +53,7 @@ The default size of a message is 5 MB. You can configure the max size of a messa ``` -> For more information on Pulsar messages, see Pulsar [binary protocol](developing-binary-protocol). +> For more information on Pulsar messages, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -171,7 +171,7 @@ Messages are received from [brokers](reference-terminology.md#broker) either syn ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -656,7 +656,7 @@ A [subscription](#subscriptions) can have one or more consumers. When a consumer #### When to use -By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry). +By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry.md). #### How to use @@ -760,7 +760,7 @@ Decisions about routing and subscription modes can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -772,7 +772,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -810,7 +810,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -881,7 +881,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -904,12 +904,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication.md). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.2.1/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.2.1/concepts-multi-tenancy.md index 8a17e72a4c8b3..26bd6a2fd183f 100644 --- a/site2/website/versioned_docs/version-2.2.1/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.2.1/concepts-multi-tenancy.md @@ -4,7 +4,7 @@ title: Multi Tenancy sidebar_label: "Multi Tenancy" --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -20,7 +20,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -28,7 +28,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.2.1/concepts-overview.md b/site2/website/versioned_docs/version-2.2.1/concepts-overview.md index 1b17c24c52658..71d5e3ff7a31a 100644 --- a/site2/website/versioned_docs/version-2.2.1/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.2.1/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Pulsar's key features include: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters * Very low publish and end-to-end latency * Seamless scalability out to over a million topics * A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md) * Multiple [subscription types](concepts-messaging.md#subscription-types) for topics ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/) -* A serverless lightweight computing framework [Pulsar Functions](functions-overview) offers stream native data processing. -* A serverless connector framework [Pulsar IO](io-overview) built on-top-of Pulsar Functions to make moving data in and out Apache Pulsar easier. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warn storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless lightweight computing framework [Pulsar Functions](functions-overview.md) offers stream native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md) built on-top-of Pulsar Functions to make moving data in and out Apache Pulsar easier. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warn storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.2.1/concepts-replication.md b/site2/website/versioned_docs/version-2.2.1/concepts-replication.md index d4653ca7aecd8..9cc3a7c3415bd 100644 --- a/site2/website/versioned_docs/version-2.2.1/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.2.1/concepts-replication.md @@ -8,7 +8,7 @@ Regardless of industries, when an unforeseen event occurs and brings day-to-day Pulsar's geo-replication mechanism is typically used for disaster recovery, enabling the replication of persistently stored message data across multiple data centers. For instance, your application is publishing data in one region and you would like to process it for consumption in other regions. With Pulsar’s geo-replication mechanism, messages can be produced and consumed in different geo-locations. -The diagram below illustrates the process of [geo-replication](administration-geo). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. +The diagram below illustrates the process of [geo-replication](administration-geo.md). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. ![A typical geo-replication example with full-mesh pattern](/assets/full-mesh-replication.svg) diff --git a/site2/website/versioned_docs/version-2.2.1/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.2.1/concepts-tiered-storage.md index f0bbde61718fb..ec031253ccd8d 100644 --- a/site2/website/versioned_docs/version-2.2.1/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.2.1/concepts-tiered-storage.md @@ -14,4 +14,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.2.1/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.2.1/concepts-topic-compaction.md index 790ce76b7c5e0..300cf68956dd1 100644 --- a/site2/website/versioned_docs/version-2.2.1/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.2.1/concepts-topic-compaction.md @@ -6,7 +6,7 @@ sidebar_label: "Topic Compaction" Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -23,7 +23,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.2.1/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.2.1/cookbooks-compaction.md index 0720019aaa035..214e899fa086d 100644 --- a/site2/website/versioned_docs/version-2.2.1/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.2.1/cookbooks-compaction.md @@ -44,7 +44,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.2.1/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.2.1/cookbooks-non-persistent.md index d40c4fbd0cf01..1346d19a04263 100644 --- a/site2/website/versioned_docs/version-2.2.1/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.2.1/cookbooks-non-persistent.md @@ -40,7 +40,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -54,7 +54,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.2.1/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.2.1/cookbooks-partitioned.md index 25896934788fc..c857d019edc02 100644 --- a/site2/website/versioned_docs/version-2.2.1/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.2.1/cookbooks-partitioned.md @@ -3,4 +3,4 @@ id: cookbooks-partitioned title: Partitioned topics sidebar_label: "Partitioned Topics" --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.2.1/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.2.1/cookbooks-retention-expiry.md index 86160c3f7ecf8..cd764fad6eb37 100644 --- a/site2/website/versioned_docs/version-2.2.1/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.2.1/cookbooks-retention-expiry.md @@ -21,7 +21,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.2.1/deploy-aws.md b/site2/website/versioned_docs/version-2.2.1/deploy-aws.md index 5db846fe790f2..60ac4c22d7672 100644 --- a/site2/website/versioned_docs/version-2.2.1/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.2.1/deploy-aws.md @@ -4,7 +4,7 @@ title: Deploying a Pulsar cluster on AWS using Terraform and Ansible sidebar_label: "Amazon Web Services" --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -209,7 +209,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.2.1/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.2.1/deploy-bare-metal-multi-cluster.md index 70dff45080b03..8a765d8843f43 100644 --- a/site2/website/versioned_docs/version-2.2.1/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.2.1/deploy-bare-metal-multi-cluster.md @@ -6,13 +6,13 @@ sidebar_label: "Bare metal multi-cluster" :::tip -1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster). -2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster.md). +2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo).Deploying a multi-cluster Pulsar instance consists of the following steps: +A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md).Deploying a multi-cluster Pulsar instance consists of the following steps: 1. Deploying two separate ZooKeeper quorums: a local quorum for each cluster in the instance and a configuration store quorum for instance-wide tasks 2. Initializing cluster metadata for each cluster @@ -21,7 +21,7 @@ A Pulsar instance consists of multiple Pulsar clusters working in unison. You ca > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. ## System requirement @@ -65,7 +65,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -244,7 +244,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -363,16 +363,16 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. -You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > **Service discovery already provided by many scheduling systems** -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.2.1/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.2.1/deploy-bare-metal.md index 04b240f43a7b4..b3d068f211c34 100644 --- a/site2/website/versioned_docs/version-2.2.1/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.2.1/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you're interested in experimenting with Pulsar or using it in a startup or on a single team, we recommend opting for a single cluster. If you do need to run a multi-cluster Pulsar instance, -however, see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +however, see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and make sure it is installed under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and make sure it is installed under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you could reference this [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you could reference this [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -273,9 +273,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (we don't recommend using a different port). -`--web-service-url-tls` | If you're using [TLS](security-tls-transport), you'll also need to specify a TLS web service URL for the cluster. The default port is 8443 (we don't recommend using a different port). +`--web-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster. The default port is 8443 (we don't recommend using a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (we don't recommend using a different port). -`--broker-service-url-tls` | If you're using [TLS](security-tls-transport), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (we don't recommend using a different port). +`--broker-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (we don't recommend using a different port). ## Deploying a BookKeeper cluster @@ -374,7 +374,7 @@ clusterName=pulsar-cluster-1 ### Enabling Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable function worker, by setting `functionsWorkerEnabled` to `true`. diff --git a/site2/website/versioned_docs/version-2.2.1/deploy-dcos.md b/site2/website/versioned_docs/version-2.2.1/deploy-dcos.md index 07f446edaffd6..599d3a1b0454c 100644 --- a/site2/website/versioned_docs/version-2.2.1/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.2.1/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -To enable all built-in [Pulsar IO](io-overview) connectors in your Pulsar deploymente, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). +To enable all built-in [Pulsar IO](io-overview.md) connectors in your Pulsar deploymente, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.2.1/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.2.1/deploy-kubernetes.md index 4e170dcd4711d..7455247c901d4 100644 --- a/site2/website/versioned_docs/version-2.2.1/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.2.1/deploy-kubernetes.md @@ -5,6 +5,6 @@ sidebar_label: "Kubernetes" --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.2.1/deploy-monitoring.md b/site2/website/versioned_docs/version-2.2.1/deploy-monitoring.md index adf3587b7aeb3..d581fe2edfacb 100644 --- a/site2/website/versioned_docs/version-2.2.1/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.2.1/deploy-monitoring.md @@ -112,7 +112,7 @@ http://$FUNCTIONS_WORKER_ADDRESS:$WORKER_PORT/metrics: You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy). +When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy.md). ## Dashboards @@ -120,7 +120,7 @@ When you collect time series statistics, the major problem is to make sure the n ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager). +The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.2.1/develop-schema.md b/site2/website/versioned_docs/version-2.2.1/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.2.1/develop-schema.md +++ b/site2/website/versioned_docs/version-2.2.1/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.2.1/functions-overview.md b/site2/website/versioned_docs/version-2.2.1/functions-overview.md index acbae672df7fd..009bc11d0d07d 100644 --- a/site2/website/versioned_docs/version-2.2.1/functions-overview.md +++ b/site2/website/versioned_docs/version-2.2.1/functions-overview.md @@ -154,7 +154,7 @@ class RoutingFunction(Function): ## Command-line interface -Pulsar Functions are managed using the [`pulsar-admin`](reference-pulsar-admin) CLI tool (in particular the [`functions`](reference-pulsar-admin.md#functions) command). Here's an example command that would run a function in [local run mode](#local-run-mode): +Pulsar Functions are managed using the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool (in particular the [`functions`](reference-pulsar-admin.md#functions) command). Here's an example command that would run a function in [local run mode](#local-run-mode): ```bash @@ -492,7 +492,7 @@ $ bin/pulsar-admin functions create \ ## Metrics -Pulsar Functions that use the [Pulsar Functions SDK](#the-pulsar-functions-sdk) can publish metrics to Pulsar. For more information, see [Metrics for Pulsar Functions](functions-metrics). +Pulsar Functions that use the [Pulsar Functions SDK](#the-pulsar-functions-sdk) can publish metrics to Pulsar. For more information, see [Metrics for Pulsar Functions](functions-metrics.md). ## State storage diff --git a/site2/website/versioned_docs/version-2.2.1/getting-started-docker.md b/site2/website/versioned_docs/version-2.2.1/getting-started-docker.md index 7e7b69f53356b..77a30ccd79f2a 100644 --- a/site2/website/versioned_docs/version-2.2.1/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.2.1/getting-started-docker.md @@ -61,8 +61,8 @@ For more info, see [Topics](concepts-messaging.md#topics). ## Start publishing and consuming messages -Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: * `pulsar://localhost:6650` @@ -117,7 +117,7 @@ client.close() ## Get the topic statistics In Pulsar you can use REST, Java, or command-line tools to control every aspect of the system. -You can find detailed documentation of all the APIs in the [Admin API Overview](admin-api-overview). +You can find detailed documentation of all the APIs in the [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.2.1/getting-started-standalone.md b/site2/website/versioned_docs/version-2.2.1/getting-started-standalone.md index d76527539cca1..24752e87792a7 100644 --- a/site2/website/versioned_docs/version-2.2.1/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.2.1/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For the purposes of local development and testing, you can run Pulsar in standalone mode on your own machine. Standalone mode includes a Pulsar broker as well as the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Run Pulsar Standalone Manually @@ -112,7 +112,7 @@ pulsar-io-twitter-@pulsar:version@.nar > If you are running Pulsar in a bare metal cluster, you need to make sure `connectors` tarball is unzipped in every broker's pulsar directory > (or in every function-worker's pulsar directory if you are running a separate worker cluster for Pulsar functions). > -> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), > you can use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ## Installing Tiered Storage Offloaders (optional) @@ -121,7 +121,7 @@ pulsar-io-twitter-@pulsar:version@.nar > If you would like to enable tiered storage feature, you can follow the instructions as below; otherwise you can > skip this section for now. -To get started using [tiered storage offloaders](concepts-tiered-storage), you'll need to download the offloaders tarball release on every broker node in +To get started using [tiered storage offloaders](concepts-tiered-storage.md), you'll need to download the offloaders tarball release on every broker node in one of the following ways: * by clicking the link below and downloading the release from an Apache mirror: @@ -161,7 +161,7 @@ For more details of how to configure tiered storage feature, you could reference > > If you are running Pulsar in a bare metal cluster, you need to make sure `offloaders` tarball is unzipped in every broker's pulsar directory > -> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), > you can use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. @@ -229,12 +229,12 @@ If the message has been successfully published to the topic, you should see a co ## Using Pulsar clients locally -Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: +Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: * `http://localhost:8080` * `pulsar://localhost:6650` -Here's an example producer for a Pulsar topic using the [Java](client-libraries-java) client: +Here's an example producer for a Pulsar topic using the [Java](client-libraries-java.md) client: ```java @@ -245,7 +245,7 @@ Producer producer = client.newProducer().topic("my-topic").create(); ``` -Here's an example [Python](client-libraries-python) producer: +Here's an example [Python](client-libraries-python.md) producer: ```python @@ -256,7 +256,7 @@ producer = client.create_producer('my-topic') ``` -Finally, here's an example [C++](client-libraries-cpp) producer: +Finally, here's an example [C++](client-libraries-cpp.md) producer: ```cpp diff --git a/site2/website/versioned_docs/version-2.2.1/io-develop.md b/site2/website/versioned_docs/version-2.2.1/io-develop.md index 69c9a2f42c428..d361a302b2529 100644 --- a/site2/website/versioned_docs/version-2.2.1/io-develop.md +++ b/site2/website/versioned_docs/version-2.2.1/io-develop.md @@ -13,15 +13,15 @@ import TabItem from '@theme/TabItem'; This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -274,7 +274,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -305,7 +305,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). @@ -380,7 +380,7 @@ Pulsar connectors enable you to move data in and out of Pulsar easily. It is imp - Check the metrics provided by Pulsar. - Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. - Set and check your customized metrics. diff --git a/site2/website/versioned_docs/version-2.2.1/io-managing.md b/site2/website/versioned_docs/version-2.2.1/io-managing.md index 47a4922b38dbd..c2e49d8f05f1a 100644 --- a/site2/website/versioned_docs/version-2.2.1/io-managing.md +++ b/site2/website/versioned_docs/version-2.2.1/io-managing.md @@ -50,7 +50,7 @@ For details, consult the documentation for [individual connectors](io-overview.m ## Running Connectors -Pulsar connectors can be managed using the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands of the [`pulsar-admin`](reference-pulsar-admin) CLI tool. +Pulsar connectors can be managed using the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands of the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. ### Running sources @@ -166,8 +166,8 @@ Here's an example to submit a Cassandra sink: ## Monitoring Connectors -Since Pulsar IO connectors are running as [Pulsar Functions](functions-overview), so you can use [`functions`](reference-pulsar-admin.md#source) commands -available in the [`pulsar-admin`](reference-pulsar-admin) CLI tool. +Since Pulsar IO connectors are running as [Pulsar Functions](functions-overview.md), so you can use [`functions`](reference-pulsar-admin.md#source) commands +available in the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. ### Retrieve Connector Metadata diff --git a/site2/website/versioned_docs/version-2.2.1/io-overview.md b/site2/website/versioned_docs/version-2.2.1/io-overview.md index f6e86827ed5ac..f501b014f6e16 100644 --- a/site2/website/versioned_docs/version-2.2.1/io-overview.md +++ b/site2/website/versioned_docs/version-2.2.1/io-overview.md @@ -8,7 +8,7 @@ original_id: io-overview Messaging systems are most powerful when you can easily use them in conjunction with external systems like databases and other messaging systems. **Pulsar IO** is a feature of Pulsar that enables you to easily create, deploy, and manage Pulsar **connectors** that interact with external systems, such as [Apache Cassandra](https://cassandra.apache.org), [Aerospike](https://www.aerospike.com), and many others. > #### Pulsar IO and Pulsar Functions -> Under the hood, Pulsar IO connectors are specialized [Pulsar Functions](functions-overview.md) purpose-built to interface with external systems. The [administrative interface](io-quickstart) for Pulsar IO is, in fact, quite similar to that of Pulsar Functions. +> Under the hood, Pulsar IO connectors are specialized [Pulsar Functions](functions-overview.md) purpose-built to interface with external systems. The [administrative interface](io-quickstart.md) for Pulsar IO is, in fact, quite similar to that of Pulsar Functions. ## Sources and sinks @@ -23,7 +23,7 @@ This diagram illustrates the relationship between sources, sinks, and Pulsar: ## Working with connectors -Pulsar IO connectors can be managed via the [`pulsar-admin`](reference-pulsar-admin) CLI tool, in particular the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands. +Pulsar IO connectors can be managed via the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool, in particular the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands. > For a guide to managing connectors in your Pulsar installation, see the [Getting started with Pulsar IO](io-quickstart.md) @@ -31,11 +31,11 @@ The following connectors are currently available for Pulsar: |Name|Java Class|Documentation| |---|---|---| -|[Aerospike sink](https://www.aerospike.com/)|[`org.apache.pulsar.io.aerospike.AerospikeSink`](https://github.com/apache/pulsar/blob/master/pulsar-io/aerospike/src/main/java/org/apache/pulsar/io/aerospike/AerospikeStringSink.java)|[Documentation](io-aerospike)| -|[Cassandra sink](https://cassandra.apache.org)|[`org.apache.pulsar.io.cassandra.CassandraSink`](https://github.com/apache/pulsar/blob/master/pulsar-io/cassandra/src/main/java/org/apache/pulsar/io/cassandra/CassandraStringSink.java)|[Documentation](io-cassandra)| +|[Aerospike sink](https://www.aerospike.com/)|[`org.apache.pulsar.io.aerospike.AerospikeSink`](https://github.com/apache/pulsar/blob/master/pulsar-io/aerospike/src/main/java/org/apache/pulsar/io/aerospike/AerospikeStringSink.java)|[Documentation](io-aerospike.md)| +|[Cassandra sink](https://cassandra.apache.org)|[`org.apache.pulsar.io.cassandra.CassandraSink`](https://github.com/apache/pulsar/blob/master/pulsar-io/cassandra/src/main/java/org/apache/pulsar/io/cassandra/CassandraStringSink.java)|[Documentation](io-cassandra.md)| |[Kafka source](https://kafka.apache.org)|[`org.apache.pulsar.io.kafka.KafkaSource`](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaStringSource.java)|[Documentation](io-kafka.md#source)| |[Kafka sink](https://kafka.apache.org)|[`org.apache.pulsar.io.kafka.KafkaSink`](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaStringSink.java)|[Documentation](io-kafka.md#sink)| |[Kinesis sink](https://aws.amazon.com/kinesis/)|[`org.apache.pulsar.io.kinesis.KinesisSink`](https://github.com/apache/pulsar/blob/master/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java)|[Documentation](io-kinesis.md#sink)| |[RabbitMQ source](https://www.rabbitmq.com)|[`org.apache.pulsar.io.rabbitmq.RabbitMQSource`](https://github.com/apache/pulsar/blob/master/pulsar-io/rabbitmq/src/main/java/org/apache/pulsar/io/rabbitmq/RabbitMQSource.java)|[Documentation](io-rabbitmq.md#sink)| |[Twitter Firehose source](https://developer.twitter.com/en/docs)|[`org.apache.pulsar.io.twitter.TwitterFireHose`](https://github.com/apache/pulsar/blob/master/pulsar-io/twitter/src/main/java/org/apache/pulsar/io/twitter/TwitterFireHose.java)|[Documentation](io-twitter.md#source)| -|[CDC Connector](https://debezium.io/)|[`org.apache.pulsar.io.kafka.connect.KafkaConnectSource`](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSource.java)|[Documentation](io-cdc)| +|[CDC Connector](https://debezium.io/)|[`org.apache.pulsar.io.kafka.connect.KafkaConnectSource`](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/KafkaConnectSource.java)|[Documentation](io-cdc.md)| diff --git a/site2/website/versioned_docs/version-2.2.1/io-quickstart.md b/site2/website/versioned_docs/version-2.2.1/io-quickstart.md index a6541d8e928ac..404c94979de21 100644 --- a/site2/website/versioned_docs/version-2.2.1/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.2.1/io-quickstart.md @@ -6,14 +6,14 @@ original_id: io-quickstart --- This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O in tandem with running the steps in this guide +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O in tandem with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you will be able to: - Connect your Pulsar cluster with your Cassandra cluster :::tip -1. These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However all +1. These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However all the commands used in this tutorial should be able to be used in a multi-nodes Pulsar cluster without any changes. 2. All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -267,11 +267,11 @@ configs: ``` -To learn more about Cassandra Connector, see [Cassandra Connector](io-cassandra). +To learn more about Cassandra Connector, see [Cassandra Connector](io-cassandra.md). ### Submit a Cassandra Sink -Pulsar provides the [CLI](reference-cli-tools) for running and managing Pulsar I/O connectors. +Pulsar provides the [CLI](reference-cli-tools.md) for running and managing Pulsar I/O connectors. We can run following command to sink a sink connector with type `cassandra` and config file `examples/cassandra-sink.yml`. @@ -296,7 +296,7 @@ as a Pulsar Function and write the messages produced in topic `test_cassandra` t ### Inspect the Cassandra Sink -Since an IO connector is running as [Pulsar Functions](functions-overview), you can use [functions CLI](reference-pulsar-admin.md#functions) +Since an IO connector is running as [Pulsar Functions](functions-overview.md), you can use [functions CLI](reference-pulsar-admin.md#functions) for inspecting and managing the IO connectors. #### Retrieve Sink Info diff --git a/site2/website/versioned_docs/version-2.2.1/pulsar-2.0.md b/site2/website/versioned_docs/version-2.2.1/pulsar-2.0.md index 560c8c1bd7cb7..07fba365e9ff2 100644 --- a/site2/website/versioned_docs/version-2.2.1/pulsar-2.0.md +++ b/site2/website/versioned_docs/version-2.2.1/pulsar-2.0.md @@ -4,13 +4,13 @@ title: Pulsar 2.0 sidebar_label: "Pulsar 2.0" --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.2.1/reference-cli-tools.md b/site2/website/versioned_docs/version-2.2.1/reference-cli-tools.md index 1c3d601e00c88..dcac11835f4b9 100644 --- a/site2/website/versioned_docs/version-2.2.1/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.2.1/reference-cli-tools.md @@ -6,7 +6,7 @@ sidebar_label: "Pulsar CLI tools" Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) @@ -825,7 +825,7 @@ Options ### `transaction` -Run a transaction. For more information, see [Pulsar transactions](txn-why). +Run a transaction. For more information, see [Pulsar transactions](txn-why.md). **Usage** diff --git a/site2/website/versioned_docs/version-2.2.1/security-athenz.md b/site2/website/versioned_docs/version-2.2.1/security-athenz.md index ba27ba43eb9d8..6d338fc4013bd 100644 --- a/site2/website/versioned_docs/version-2.2.1/security-athenz.md +++ b/site2/website/versioned_docs/version-2.2.1/security-athenz.md @@ -75,7 +75,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.2.1/security-authorization.md b/site2/website/versioned_docs/version-2.2.1/security-authorization.md index d9d33665d066b..c5e3c04b03524 100644 --- a/site2/website/versioned_docs/version-2.2.1/security-authorization.md +++ b/site2/website/versioned_docs/version-2.2.1/security-authorization.md @@ -26,7 +26,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -58,7 +58,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -86,7 +86,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication @@ -116,7 +116,7 @@ PulsarAdmin admin = PulsarAdmin.builder() When a client is identified with multiple roles in a token (the type of role claim in the token is an array) during the authentication process, Pulsar supports to check the permissions of all the roles and further authorize the client as long as one of its roles has the required permissions. > **Note**
    -> This authorization method is only compatible with [JWT authentication](security-jwt). +> This authorization method is only compatible with [JWT authentication](security-jwt.md). To enable this authorization method, configure the authorization provider as `MultiRolesTokenAuthorizationProvider` in the `conf/broker.conf` file. diff --git a/site2/website/versioned_docs/version-2.2.1/security-overview.md b/site2/website/versioned_docs/version-2.2.1/security-overview.md index 2f3fc035ad4cb..3f08cb29ad854 100644 --- a/site2/website/versioned_docs/version-2.2.1/security-overview.md +++ b/site2/website/versioned_docs/version-2.2.1/security-overview.md @@ -20,7 +20,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.2.1/security-tls-authentication.md b/site2/website/versioned_docs/version-2.2.1/security-tls-authentication.md index bbc0514afa9ac..127140dc0c8be 100644 --- a/site2/website/versioned_docs/version-2.2.1/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.2.1/security-tls-authentication.md @@ -7,7 +7,7 @@ original_id: security-tls-authentication ## TLS Authentication Overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport), but instead of only servers having keys and certs which the client uses to verify the server's identity, clients also have keys and certs which the server uses to verify the client's identity. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md), but instead of only servers having keys and certs which the client uses to verify the server's identity, clients also have keys and certs which the server uses to verify the client's identity. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. ### Creating client certificates @@ -80,7 +80,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, put the following in `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. The role token for this key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. The role token for this key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -100,7 +100,7 @@ When TLS authentication, the client needs to connect via TLS transport, so you n ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You'll need to add the following parameters to that file to use TLS authentication with Pulsar's CLI tools: diff --git a/site2/website/versioned_docs/version-2.2.1/security-tls-transport.md b/site2/website/versioned_docs/version-2.2.1/security-tls-transport.md index d4a36de0a3dcd..51e90aa766080 100644 --- a/site2/website/versioned_docs/version-2.2.1/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.2.1/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text, which means that all data is sent in the clear. TLS can be used to encrypt this traffic so that it cannot be snooped by a man-in-the-middle attacker. -TLS can be configured for both encryption and authentication. You may configure just TLS transport encryption, which is covered in this guide. TLS authentication is covered [elsewhere](security-tls-authentication.md). Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +TLS can be configured for both encryption and authentication. You may configure just TLS transport encryption, which is covered in this guide. TLS authentication is covered [elsewhere](security-tls-authentication.md). Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may have a performance impact due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -A third kind of key pair, **client key pairs**, are used for [client authentication](security-tls-authentication). +A third kind of key pair, **client key pairs**, are used for [client authentication](security-tls-authentication.md). The **certificate authority** private key should be stored in a very secure location (a fully encrypted, disconnected, air gapped computer). The certificate authority public key, the **trust cert**, can be freely shared. @@ -27,7 +27,7 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server they are talking to has a key pair that was signed by the certificate authority. A man-in-the-middle attacker would not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that was signed by the certificate authority. The Common Name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that was signed by the certificate authority. The Common Name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). ## Creating TLS Certificates @@ -128,7 +128,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Broker Configuration -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you'll need to make some changes to `broker.conf`, which is located in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you'll need to make some changes to `broker.conf`, which is located in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -199,7 +199,7 @@ The examples below show hostname verification being disabled for the Java client ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You'll need to add the following parameters to that file to use TLS transport with Pulsar's CLI tools: diff --git a/site2/website/versioned_docs/version-2.2.1/security-token-admin.md b/site2/website/versioned_docs/version-2.2.1/security-token-admin.md index f3cec69ec9d35..60daefefeded1 100644 --- a/site2/website/versioned_docs/version-2.2.1/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.2.1/security-token-admin.md @@ -171,7 +171,7 @@ brokerClientAuthenticationParameters=token:eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ0ZXN0 To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.2.1/security-token-client.md b/site2/website/versioned_docs/version-2.2.1/security-token-client.md index 3af82b476073e..d0ebd0168566d 100644 --- a/site2/website/versioned_docs/version-2.2.1/security-token-client.md +++ b/site2/website/versioned_docs/version-2.2.1/security-token-client.md @@ -26,12 +26,12 @@ Application will specify the token when creating the client instance. An alterna a "token supplier", that is to say a function that returns the token when the client library will need one. -See [Token authentication admin](security-token-admin) for a reference on how to enable token +See [Token authentication admin](security-token-admin.md) for a reference on how to enable token authentication on a Pulsar cluster. ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You'll need to add the following parameters to that file to use the token authentication with Pulsar's CLI tools: diff --git a/site2/website/versioned_docs/version-2.2.1/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.2.1/sql-deployment-configurations.md index 639ac988b3409..069c6ac9c92da 100644 --- a/site2/website/versioned_docs/version-2.2.1/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.2.1/sql-deployment-configurations.md @@ -141,7 +141,7 @@ Since Pulsar SQL is powered by [Trino (formerly Presto SQL)](https://trino.io), :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.2.1/sql-overview.md b/site2/website/versioned_docs/version-2.2.1/sql-overview.md index 2f827f4267933..5b01dfde56e7f 100644 --- a/site2/website/versioned_docs/version-2.2.1/sql-overview.md +++ b/site2/website/versioned_docs/version-2.2.1/sql-overview.md @@ -4,7 +4,7 @@ title: Pulsar SQL Overview sidebar_label: "Overview" --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL)](https://trino.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL.md)](https://trino.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.2.1/standalone-docker.md b/site2/website/versioned_docs/version-2.2.1/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.2.1/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.2.1/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.2.1/standalone.md b/site2/website/versioned_docs/version-2.2.1/standalone.md index c17a49306cd40..6cc589f5f1241 100644 --- a/site2/website/versioned_docs/version-2.2.1/standalone.md +++ b/site2/website/versioned_docs/version-2.2.1/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For the purposes of local development and testing, you can run Pulsar in standalone mode on your own machine. Standalone mode includes a Pulsar broker as well as the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Run Pulsar Standalone Manually @@ -112,7 +112,7 @@ pulsar-io-twitter-@pulsar:version@.nar > If you are running Pulsar in a bare metal cluster, you need to make sure `connectors` tarball is unzipped in every broker's pulsar directory > (or in every function-worker's pulsar directory if you are running a separate worker cluster for Pulsar functions). > -> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), > you can use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ## Installing Tiered Storage Offloaders (optional) @@ -121,7 +121,7 @@ pulsar-io-twitter-@pulsar:version@.nar > If you would like to enable tiered storage feature, you can follow the instructions as below; otherwise you can > skip this section for now. -To get started using [tiered storage offloaders](concepts-tiered-storage), you'll need to download the offloaders tarball release on every broker node in +To get started using [tiered storage offloaders](concepts-tiered-storage.md), you'll need to download the offloaders tarball release on every broker node in one of the following ways: * by clicking the link below and downloading the release from an Apache mirror: @@ -161,7 +161,7 @@ For more details of how to configure tiered storage feature, you could reference > > If you are running Pulsar in a bare metal cluster, you need to make sure `offloaders` tarball is unzipped in every broker's pulsar directory > -> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), > you can use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. @@ -229,12 +229,12 @@ If the message has been successfully published to the topic, you should see a co ## Using Pulsar clients locally -Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: +Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: * `http://localhost:8080` * `pulsar://localhost:6650` -Here's an example producer for a Pulsar topic using the [Java](client-libraries-java) client: +Here's an example producer for a Pulsar topic using the [Java](client-libraries-java.md) client: ```java @@ -245,7 +245,7 @@ Producer producer = client.newProducer().topic("my-topic").create(); ``` -Here's an example [Python](client-libraries-python) producer: +Here's an example [Python](client-libraries-python.md) producer: ```python @@ -256,7 +256,7 @@ producer = client.create_producer('my-topic') ``` -Finally, here's an example [C++](client-libraries-cpp) producer: +Finally, here's an example [C++](client-libraries-cpp.md) producer: ```cpp diff --git a/site2/website/versioned_docs/version-2.3.0/admin-api-clusters.md b/site2/website/versioned_docs/version-2.3.0/admin-api-clusters.md index 04db2f60759bc..1255ad25d7aac 100644 --- a/site2/website/versioned_docs/version-2.3.0/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.3.0/admin-api-clusters.md @@ -88,7 +88,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -112,11 +112,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block > - For the latest and complete information about `Java admin API`, including classes, methods, descriptions, and more, see [Java admin API doc](https://pulsar.apache.org/api/admin/). -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -121,7 +121,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block For a guide to managing message persistence, retention, and expiry in Pulsar, see [this cookbook](cookbooks-retention-expiry). +> For a guide to managing message persistence, retention, and expiry in Pulsar, see [this cookbook](cookbooks-retention-expiry.md). ### Deploying BookKeeper diff --git a/site2/website/versioned_docs/version-2.3.0/client-libraries-go.md b/site2/website/versioned_docs/version-2.3.0/client-libraries-go.md index 9e33717d556ee..6720cbf6877da 100644 --- a/site2/website/versioned_docs/version-2.3.0/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.3.0/client-libraries-go.md @@ -16,7 +16,7 @@ The Pulsar Go client can be used to create Pulsar [producers](#producers), [cons ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Installing go package @@ -50,7 +50,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -68,7 +68,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -223,14 +223,14 @@ Parameter | Description | Default :---------|:------------|:------- `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | | `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/) and [`ZSTD`](https://facebook.github.io/zstd/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | ## Consumers @@ -475,7 +475,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.3.0/client-libraries-java.md b/site2/website/versioned_docs/version-2.3.0/client-libraries-java.md index 2c349dbf12f12..1058b0896fa78 100644 --- a/site2/website/versioned_docs/version-2.3.0/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.3.0/client-libraries-java.md @@ -10,7 +10,7 @@ import TabItem from '@theme/TabItem'; ```` -You can use a Pulsar Java client to create the Java [producer](#producer), [consumer](#consumer), [reader](#reader) and [TableView](#tableview) of messages and to perform [administrative tasks](admin-api-overview). The current Java client version is **@pulsar:version@**. +You can use a Pulsar Java client to create the Java [producer](#producer), [consumer](#consumer), [reader](#reader) and [TableView](#tableview) of messages and to perform [administrative tasks](admin-api-overview.md). The current Java client version is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), [reader](#reader) and [TableView](#tableview) of a Java client are thread-safe. @@ -19,10 +19,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | [The producer and consumer API](https://pulsar.apache.org/api/client/) | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Include both `pulsar-client` and `pulsar-client-admin`
    Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.
    In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -69,7 +69,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -95,7 +95,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -126,7 +126,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -279,7 +279,7 @@ Currently, cluster-level failover can perform probes to prevent data loss, but i > #### What are the relationships between cluster-level failover and geo-replication? -The cluster-level failover is an extension of [geo-replication](concepts-replication) to improve stability and robustness. The cluster-level failover depends on geo-replication, and they have some **differences** as below. +The cluster-level failover is an extension of [geo-replication](concepts-replication.md) to improve stability and robustness. The cluster-level failover depends on geo-replication, and they have some **differences** as below. Influence |Cluster-level failover|Geo-replication |---|---|--- @@ -576,7 +576,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics cookbook](cookbooks-partitioned). +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics cookbook](cookbooks-partitioned.md). ### Async send @@ -1370,7 +1370,7 @@ tv.forEach((key, value) -> /*operations on all existing messages*/) ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producer) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producer) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -1488,7 +1488,7 @@ For example of ProtobufNativeSchema, see [`SchemaDefinition` in `Complex type`]( ## Authentication -Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2). You can use the Pulsar Java client with all of them. +Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2.md). You can use the Pulsar Java client with all of them. ### TLS Authentication @@ -1515,7 +1515,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` @@ -1552,7 +1552,7 @@ PulsarClient client = PulsarClient.builder() ### Oauth2 -The following example shows how to use [Oauth2](security-oauth2) as an authentication provider for the Pulsar Java client. +The following example shows how to use [Oauth2](security-oauth2.md) as an authentication provider for the Pulsar Java client. You can use the factory method to configure authentication for Pulsar Java client. diff --git a/site2/website/versioned_docs/version-2.3.0/client-libraries-python.md b/site2/website/versioned_docs/version-2.3.0/client-libraries-python.md index 02b1f498d2f06..f0a8f92304ff4 100644 --- a/site2/website/versioned_docs/version-2.3.0/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.3.0/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -The Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +The Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. ## Installation diff --git a/site2/website/versioned_docs/version-2.3.0/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.3.0/client-libraries-websocket.md index a6e60361749de..0b31ffe78161a 100644 --- a/site2/website/versioned_docs/version-2.3.0/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.3.0/client-libraries-websocket.md @@ -4,14 +4,14 @@ title: Pulsar WebSocket API sidebar_label: "WebSocket" --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.3.0/client-libraries.md b/site2/website/versioned_docs/version-2.3.0/client-libraries.md index 536cd0ccda82f..8f91dfbbbdd8c 100644 --- a/site2/website/versioned_docs/version-2.3.0/client-libraries.md +++ b/site2/website/versioned_docs/version-2.3.0/client-libraries.md @@ -8,12 +8,12 @@ Pulsar supports the following client libraries: |Language|Documentation|Release note|Code repo |---|---|---|--- -Java |- [User doc](client-libraries-java)

    - [API doc](https://pulsar.apache.org/api/client/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client) -C++ | - [User doc](client-libraries-cpp)

    - [API doc](https://pulsar.apache.org/api/cpp/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp) -Python | - [User doc](client-libraries-python)

    - [API doc](https://pulsar.apache.org/api/python/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) -WebSocket| [User doc](client-libraries-websocket) | [Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-websocket) +Java |- [User doc](client-libraries-java.md)

    - [API doc](https://pulsar.apache.org/api/client/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client) +C++ | - [User doc](client-libraries-cpp.md)

    - [API doc](https://pulsar.apache.org/api/cpp/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp) +Python | - [User doc](client-libraries-python.md)

    - [API doc](https://pulsar.apache.org/api/python/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) +WebSocket| [User doc](client-libraries-websocket.md) | [Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-websocket) Go client|[User doc](client-libraries-go.md)|[Here](https://github.com/apache/pulsar-client-go/blob/master/CHANGELOG) |[Here](https://github.com/apache/pulsar-client-go) -Node.js|[User doc](client-libraries-node)|[Here](https://github.com/apache/pulsar-client-node/releases) |[Here](https://github.com/apache/pulsar-client-node) +Node.js|[User doc](client-libraries-node.md)|[Here](https://github.com/apache/pulsar-client-node/releases) |[Here](https://github.com/apache/pulsar-client-node) C# |[User doc](client-libraries-dotnet.md)| [Here](https://github.com/apache/pulsar-dotpulsar/blob/master/CHANGELOG)|[Here](https://github.com/apache/pulsar-dotpulsar) :::note diff --git a/site2/website/versioned_docs/version-2.3.0/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.3.0/concepts-architecture-overview.md index 0d96b59f8ae91..1ce346fc16941 100644 --- a/site2/website/versioned_docs/version-2.3.0/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.3.0/concepts-architecture-overview.md @@ -4,7 +4,7 @@ title: Architecture Overview sidebar_label: "Architecture" --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -16,20 +16,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers. The producers connect to the brokers to publish messages and the consumers connect to the brokers to consume the messages. -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -39,9 +39,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -139,17 +139,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -157,7 +157,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.3.0/concepts-authentication.md b/site2/website/versioned_docs/version-2.3.0/concepts-authentication.md index 335da8dc5a519..fbefead69f382 100644 --- a/site2/website/versioned_docs/version-2.3.0/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.3.0/concepts-authentication.md @@ -4,5 +4,5 @@ title: Authentication and Authorization sidebar_label: "Authentication and Authorization" --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.3.0/concepts-clients.md b/site2/website/versioned_docs/version-2.3.0/concepts-clients.md index 65201b5ce1cd2..c0fe61cd06e76 100644 --- a/site2/website/versioned_docs/version-2.3.0/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.3.0/concepts-clients.md @@ -4,12 +4,12 @@ title: Pulsar Clients sidebar_label: "Clients" --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -37,7 +37,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.3.0/concepts-messaging.md b/site2/website/versioned_docs/version-2.3.0/concepts-messaging.md index a49105fa76954..b43e325b0e212 100644 --- a/site2/website/versioned_docs/version-2.3.0/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.3.0/concepts-messaging.md @@ -24,7 +24,7 @@ Publish time | The timestamp of when the message was published (automatically ap Event time | An optional timestamp that applications can attach to the message representing when something happened, e.g. when the message was processed. The event time of a message is 0 if none is explicitly set. -> For a more in-depth breakdown of Pulsar message contents, see the documentation on Pulsar's [binary protocol](developing-binary-protocol). +> For a more in-depth breakdown of Pulsar message contents, see the documentation on Pulsar's [binary protocol](developing-binary-protocol.md). ## Producers @@ -75,7 +75,7 @@ Messages can be acknowledged either one by one or cumulatively. With cumulative ### Listeners -Client libraries can provide their own listener implementations for consumers. The [Java client](client-libraries-java), for example, provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries can provide their own listener implementations for consumers. The [Java client](client-libraries-java.md), for example, provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ## Topics @@ -200,7 +200,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -212,7 +212,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -250,7 +250,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -298,7 +298,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -321,12 +321,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, which means that you don't need to modify your Pulsar client code. Instead, you only need to make administrative changes (see the [Managing message deduplication](cookbooks-deduplication) cookbook for a guide). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, which means that you don't need to modify your Pulsar client code. Instead, you only need to make administrative changes (see the [Managing message deduplication](cookbooks-deduplication.md) cookbook for a guide). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.3.0/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.3.0/concepts-multi-tenancy.md index 8a17e72a4c8b3..26bd6a2fd183f 100644 --- a/site2/website/versioned_docs/version-2.3.0/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.3.0/concepts-multi-tenancy.md @@ -4,7 +4,7 @@ title: Multi Tenancy sidebar_label: "Multi Tenancy" --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -20,7 +20,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -28,7 +28,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.3.0/concepts-overview.md b/site2/website/versioned_docs/version-2.3.0/concepts-overview.md index c9e9db298d003..f238e14b05da7 100644 --- a/site2/website/versioned_docs/version-2.3.0/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.3.0/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Pulsar's key features include: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters * Very low publish and end-to-end latency * Seamless scalability out to over a million topics * A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md) * Multiple [subscription types](concepts-messaging.md#subscription-types) for topics ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/) -* A serverless lightweight computing framework [Pulsar Functions](functions-overview) offers stream native data processing. -* A serverless connector framework [Pulsar IO](io-overview) built on-top-of Pulsar Functions to make moving data in and out Apache Pulsar easier. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless lightweight computing framework [Pulsar Functions](functions-overview.md) offers stream native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md) built on-top-of Pulsar Functions to make moving data in and out Apache Pulsar easier. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.3.0/concepts-replication.md b/site2/website/versioned_docs/version-2.3.0/concepts-replication.md index d4653ca7aecd8..9cc3a7c3415bd 100644 --- a/site2/website/versioned_docs/version-2.3.0/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.3.0/concepts-replication.md @@ -8,7 +8,7 @@ Regardless of industries, when an unforeseen event occurs and brings day-to-day Pulsar's geo-replication mechanism is typically used for disaster recovery, enabling the replication of persistently stored message data across multiple data centers. For instance, your application is publishing data in one region and you would like to process it for consumption in other regions. With Pulsar’s geo-replication mechanism, messages can be produced and consumed in different geo-locations. -The diagram below illustrates the process of [geo-replication](administration-geo). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. +The diagram below illustrates the process of [geo-replication](administration-geo.md). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. ![A typical geo-replication example with full-mesh pattern](/assets/full-mesh-replication.svg) diff --git a/site2/website/versioned_docs/version-2.3.0/concepts-schema-registry.md b/site2/website/versioned_docs/version-2.3.0/concepts-schema-registry.md index 1c2ff47d3482e..a4c8ae8eb81e9 100644 --- a/site2/website/versioned_docs/version-2.3.0/concepts-schema-registry.md +++ b/site2/website/versioned_docs/version-2.3.0/concepts-schema-registry.md @@ -17,7 +17,7 @@ Both approaches are available in Pulsar, and you're free to adopt one or the oth #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ## Basic architecture @@ -39,7 +39,7 @@ Pulsar schemas are fairly simple data structures that consist of: ## Schema versions -In order to illustrate how schema versioning works, let's walk through an example. Imagine that the Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begin sending messages: +In order to illustrate how schema versioning works, let's walk through an example. Imagine that the Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begin sending messages: ```java diff --git a/site2/website/versioned_docs/version-2.3.0/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.3.0/concepts-tiered-storage.md index f0bbde61718fb..ec031253ccd8d 100644 --- a/site2/website/versioned_docs/version-2.3.0/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.3.0/concepts-tiered-storage.md @@ -14,4 +14,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.3.0/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.3.0/concepts-topic-compaction.md index 790ce76b7c5e0..300cf68956dd1 100644 --- a/site2/website/versioned_docs/version-2.3.0/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.3.0/concepts-topic-compaction.md @@ -6,7 +6,7 @@ sidebar_label: "Topic Compaction" Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -23,7 +23,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.3.0/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.3.0/cookbooks-compaction.md index 0720019aaa035..214e899fa086d 100644 --- a/site2/website/versioned_docs/version-2.3.0/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.3.0/cookbooks-compaction.md @@ -44,7 +44,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.3.0/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.3.0/cookbooks-non-persistent.md index d40c4fbd0cf01..1346d19a04263 100644 --- a/site2/website/versioned_docs/version-2.3.0/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.3.0/cookbooks-non-persistent.md @@ -40,7 +40,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -54,7 +54,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.3.0/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.3.0/cookbooks-partitioned.md index 25896934788fc..c857d019edc02 100644 --- a/site2/website/versioned_docs/version-2.3.0/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.3.0/cookbooks-partitioned.md @@ -3,4 +3,4 @@ id: cookbooks-partitioned title: Partitioned topics sidebar_label: "Partitioned Topics" --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.3.0/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.3.0/cookbooks-retention-expiry.md index db4c4fe4c1eef..64e1f405fc239 100644 --- a/site2/website/versioned_docs/version-2.3.0/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.3.0/cookbooks-retention-expiry.md @@ -15,7 +15,7 @@ In Pulsar, you can override both of these default behaviors, at the namespace le * You can persistently store messages that have already been consumed and acknowledged for a minimum time by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe, can be automatically marked as consumed, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL at the namespace level (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL at the namespace level (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL are solving two different problems diff --git a/site2/website/versioned_docs/version-2.3.0/deploy-aws.md b/site2/website/versioned_docs/version-2.3.0/deploy-aws.md index 5db846fe790f2..60ac4c22d7672 100644 --- a/site2/website/versioned_docs/version-2.3.0/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.3.0/deploy-aws.md @@ -4,7 +4,7 @@ title: Deploying a Pulsar cluster on AWS using Terraform and Ansible sidebar_label: "Amazon Web Services" --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -209,7 +209,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.3.0/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.3.0/deploy-bare-metal-multi-cluster.md index 70dff45080b03..8a765d8843f43 100644 --- a/site2/website/versioned_docs/version-2.3.0/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.3.0/deploy-bare-metal-multi-cluster.md @@ -6,13 +6,13 @@ sidebar_label: "Bare metal multi-cluster" :::tip -1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster). -2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster.md). +2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo).Deploying a multi-cluster Pulsar instance consists of the following steps: +A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md).Deploying a multi-cluster Pulsar instance consists of the following steps: 1. Deploying two separate ZooKeeper quorums: a local quorum for each cluster in the instance and a configuration store quorum for instance-wide tasks 2. Initializing cluster metadata for each cluster @@ -21,7 +21,7 @@ A Pulsar instance consists of multiple Pulsar clusters working in unison. You ca > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. ## System requirement @@ -65,7 +65,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -244,7 +244,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -363,16 +363,16 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. -You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > **Service discovery already provided by many scheduling systems** -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.3.0/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.3.0/deploy-bare-metal.md index ace80182ddd7e..5b3bae38c599e 100644 --- a/site2/website/versioned_docs/version-2.3.0/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.3.0/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you're interested in experimenting with Pulsar or using it in a startup or on a single team, we recommend opting for a single cluster. If you do need to run a multi-cluster Pulsar instance, -however, see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +however, see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and make sure it is installed under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and make sure it is installed under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you could reference this [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you could reference this [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -264,9 +264,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (we don't recommend using a different port). -`--web-service-url-tls` | If you're using [TLS](security-tls-transport), you'll also need to specify a TLS web service URL for the cluster. The default port is 8443 (we don't recommend using a different port). +`--web-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster. The default port is 8443 (we don't recommend using a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (we don't recommend using a different port). -`--broker-service-url-tls` | If you're using [TLS](security-tls-transport), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (we don't recommend using a different port). +`--broker-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (we don't recommend using a different port). ## Deploying a BookKeeper cluster @@ -365,7 +365,7 @@ clusterName=pulsar-cluster-1 ### Enabling Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable function worker, by setting `functionsWorkerEnabled` to `true`. diff --git a/site2/website/versioned_docs/version-2.3.0/deploy-dcos.md b/site2/website/versioned_docs/version-2.3.0/deploy-dcos.md index 07f446edaffd6..599d3a1b0454c 100644 --- a/site2/website/versioned_docs/version-2.3.0/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.3.0/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -To enable all built-in [Pulsar IO](io-overview) connectors in your Pulsar deploymente, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). +To enable all built-in [Pulsar IO](io-overview.md) connectors in your Pulsar deploymente, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.3.0/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.3.0/deploy-kubernetes.md index 4e170dcd4711d..7455247c901d4 100644 --- a/site2/website/versioned_docs/version-2.3.0/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.3.0/deploy-kubernetes.md @@ -5,6 +5,6 @@ sidebar_label: "Kubernetes" --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.3.0/deploy-monitoring.md b/site2/website/versioned_docs/version-2.3.0/deploy-monitoring.md index 677c9092bd2cb..97f60cd17aa31 100644 --- a/site2/website/versioned_docs/version-2.3.0/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.3.0/deploy-monitoring.md @@ -89,7 +89,7 @@ For that reason we only collect time series of metrics aggregated at the namespa ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Dashboard](administration-dashboard). +The per-topic dashboard instructions are available at [Dashboard](administration-dashboard.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.3.0/develop-schema.md b/site2/website/versioned_docs/version-2.3.0/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.3.0/develop-schema.md +++ b/site2/website/versioned_docs/version-2.3.0/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.3.0/functions-deploying.md b/site2/website/versioned_docs/version-2.3.0/functions-deploying.md index 19565c1c4389c..d5395324ef864 100644 --- a/site2/website/versioned_docs/version-2.3.0/functions-deploying.md +++ b/site2/website/versioned_docs/version-2.3.0/functions-deploying.md @@ -19,8 +19,8 @@ Cluster mode | The function runs *inside of* your Pulsar cluster, on the same ma In order to deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more If you're running a non-[standalone](reference-terminology.md#standalone) cluster, you'll need to obtain the service URL for the cluster. How you obtain the service URL will depend on how you deployed your Pulsar cluster. @@ -52,7 +52,7 @@ Function name | Whichever value is specified for the class name (minus org, libr Tenant | Derived from the input topics' names. If the input topics are under the `marketing` tenant---i.e. the topic names have the form `persistent://marketing/{namespace}/{topicName}`---then the tenant will be `marketing`. Namespace | Derived from the input topics' names. If the input topics are under the `asia` namespace under the `marketing` tenant---i.e. the topic names have the form `persistent://marketing/asia/{topicName}`, then the namespace will be `asia`. Output topic | `{input topic}-{function name}-output`. A function with an input topic name of `incoming` and a function name of `exclamation`, for example, would have an output topic of `incoming-exclamation-output`. -Subscription type | For at-least-once and at-most-once [processing guarantees](functions-guarantees), the [`SHARED`](concepts-messaging.md#shared) is applied by default; for effectively-once guarantees, [`FAILOVER`](concepts-messaging.md#failover) is applied +Subscription type | For at-least-once and at-most-once [processing guarantees](functions-guarantees.md), the [`SHARED`](concepts-messaging.md#shared) is applied by default; for effectively-once guarantees, [`FAILOVER`](concepts-messaging.md#failover) is applied Processing guarantees | [`ATLEAST_ONCE`](functions-guarantees.md) Pulsar service URL | `pulsar://localhost:6650` diff --git a/site2/website/versioned_docs/version-2.3.0/functions-overview.md b/site2/website/versioned_docs/version-2.3.0/functions-overview.md index c84be36c58882..3a59d43923e09 100644 --- a/site2/website/versioned_docs/version-2.3.0/functions-overview.md +++ b/site2/website/versioned_docs/version-2.3.0/functions-overview.md @@ -163,7 +163,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.3.0/getting-started-docker.md b/site2/website/versioned_docs/version-2.3.0/getting-started-docker.md index 26908245cb2c9..904d67401668a 100644 --- a/site2/website/versioned_docs/version-2.3.0/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.3.0/getting-started-docker.md @@ -43,8 +43,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -103,7 +103,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.3.0/getting-started-standalone.md b/site2/website/versioned_docs/version-2.3.0/getting-started-standalone.md index e82283b002d6a..f6647488180bc 100644 --- a/site2/website/versioned_docs/version-2.3.0/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.3.0/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For the purposes of local development and testing, you can run Pulsar in standalone mode on your own machine. Standalone mode includes a Pulsar broker as well as the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Run Pulsar Standalone Manually @@ -103,7 +103,7 @@ pulsar-io-aerospike-@pulsar:version@.nar > If you are running Pulsar in a bare metal cluster, you need to make sure `connectors` tarball is unzipped in every broker's pulsar directory > (or in every function-worker's pulsar directory if you are running a separate worker cluster for Pulsar functions). > -> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), > you can use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ## Installing Tiered Storage Offloaders (optional) @@ -112,7 +112,7 @@ pulsar-io-aerospike-@pulsar:version@.nar > If you would like to enable tiered storage feature, you can follow the instructions as below; otherwise you can > skip this section for now. -To get started using [tiered storage offloaders](concepts-tiered-storage), you'll need to download the offloaders tarball release on every broker node in +To get started using [tiered storage offloaders](concepts-tiered-storage.md), you'll need to download the offloaders tarball release on every broker node in one of the following ways: * by clicking the link below and downloading the release from an Apache mirror: @@ -152,7 +152,7 @@ For more details of how to configure tiered storage feature, you could reference > > If you are running Pulsar in a bare metal cluster, you need to make sure `offloaders` tarball is unzipped in every broker's pulsar directory > -> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), > you can use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. @@ -239,12 +239,12 @@ hello-pulsar ## Using Pulsar clients locally -Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: +Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: * `http://localhost:8080` * `pulsar://localhost:6650` -Here's an example producer for a Pulsar topic using the [Java](client-libraries-java) client: +Here's an example producer for a Pulsar topic using the [Java](client-libraries-java.md) client: ```java @@ -255,7 +255,7 @@ Producer producer = client.newProducer().topic("my-topic").create(); ``` -Here's an example [Python](client-libraries-python) producer: +Here's an example [Python](client-libraries-python.md) producer: ```python @@ -266,7 +266,7 @@ producer = client.create_producer('my-topic') ``` -Finally, here's an example [C++](client-libraries-cpp) producer: +Finally, here's an example [C++](client-libraries-cpp.md) producer: ```cpp diff --git a/site2/website/versioned_docs/version-2.3.0/io-develop.md b/site2/website/versioned_docs/version-2.3.0/io-develop.md index 69c9a2f42c428..d361a302b2529 100644 --- a/site2/website/versioned_docs/version-2.3.0/io-develop.md +++ b/site2/website/versioned_docs/version-2.3.0/io-develop.md @@ -13,15 +13,15 @@ import TabItem from '@theme/TabItem'; This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -274,7 +274,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -305,7 +305,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). @@ -380,7 +380,7 @@ Pulsar connectors enable you to move data in and out of Pulsar easily. It is imp - Check the metrics provided by Pulsar. - Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. - Set and check your customized metrics. diff --git a/site2/website/versioned_docs/version-2.3.0/io-managing.md b/site2/website/versioned_docs/version-2.3.0/io-managing.md index cce795bbae86a..8c834410a5d0f 100644 --- a/site2/website/versioned_docs/version-2.3.0/io-managing.md +++ b/site2/website/versioned_docs/version-2.3.0/io-managing.md @@ -50,7 +50,7 @@ For details, consult the documentation for [individual connectors](io-overview.m ## Running Connectors -Pulsar connectors can be managed using the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands of the [`pulsar-admin`](reference-pulsar-admin) CLI tool. +Pulsar connectors can be managed using the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands of the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. ### Running sources @@ -166,8 +166,8 @@ Here's an example to submit a Cassandra sink: ## Monitoring Connectors -Since Pulsar IO connectors are running as [Pulsar Functions](functions-overview), so you can use [`functions`](reference-pulsar-admin.md#source) commands -available in the [`pulsar-admin`](reference-pulsar-admin) CLI tool. +Since Pulsar IO connectors are running as [Pulsar Functions](functions-overview.md), so you can use [`functions`](reference-pulsar-admin.md#source) commands +available in the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. ### Retrieve Connector Metadata diff --git a/site2/website/versioned_docs/version-2.3.0/io-overview.md b/site2/website/versioned_docs/version-2.3.0/io-overview.md index 2a6ecaee5883f..ea49e4b042e8c 100644 --- a/site2/website/versioned_docs/version-2.3.0/io-overview.md +++ b/site2/website/versioned_docs/version-2.3.0/io-overview.md @@ -157,7 +157,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.3.0/io-quickstart.md b/site2/website/versioned_docs/version-2.3.0/io-quickstart.md index c4230323e66f4..83a57170dfc91 100644 --- a/site2/website/versioned_docs/version-2.3.0/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.3.0/io-quickstart.md @@ -6,14 +6,14 @@ original_id: io-quickstart --- This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O in tandem with running the steps in this guide +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O in tandem with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you will be able to: - Connect your Pulsar cluster with your Cassandra cluster :::tip -1. These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However all +1. These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However all the commands used in this tutorial should be able to be used in a multi-nodes Pulsar cluster without any changes. 2. All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -260,11 +260,11 @@ configs: ``` -To learn more about Cassandra Connector, see [Cassandra Connector](io-cassandra). +To learn more about Cassandra Connector, see [Cassandra Connector](io-cassandra.md). ### Submit a Cassandra Sink -Pulsar provides the [CLI](reference-cli-tools) for running and managing Pulsar I/O connectors. +Pulsar provides the [CLI](reference-cli-tools.md) for running and managing Pulsar I/O connectors. We can run following command to sink a sink connector with type `cassandra` and config file `examples/cassandra-sink.yml`. diff --git a/site2/website/versioned_docs/version-2.3.0/pulsar-2.0.md b/site2/website/versioned_docs/version-2.3.0/pulsar-2.0.md index 560c8c1bd7cb7..07fba365e9ff2 100644 --- a/site2/website/versioned_docs/version-2.3.0/pulsar-2.0.md +++ b/site2/website/versioned_docs/version-2.3.0/pulsar-2.0.md @@ -4,13 +4,13 @@ title: Pulsar 2.0 sidebar_label: "Pulsar 2.0" --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.3.0/reference-cli-tools.md b/site2/website/versioned_docs/version-2.3.0/reference-cli-tools.md index 204079aa6ce1b..4e8c146f68e33 100644 --- a/site2/website/versioned_docs/version-2.3.0/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.3.0/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.3.0/security-athenz.md b/site2/website/versioned_docs/version-2.3.0/security-athenz.md index ba27ba43eb9d8..6d338fc4013bd 100644 --- a/site2/website/versioned_docs/version-2.3.0/security-athenz.md +++ b/site2/website/versioned_docs/version-2.3.0/security-athenz.md @@ -75,7 +75,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.3.0/security-authorization.md b/site2/website/versioned_docs/version-2.3.0/security-authorization.md index 1f706a56b5abe..bbb0a049d2bcd 100644 --- a/site2/website/versioned_docs/version-2.3.0/security-authorization.md +++ b/site2/website/versioned_docs/version-2.3.0/security-authorization.md @@ -29,7 +29,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters available in the `conf/broker.conf` file, > as well as the default values for those parameters, can be found in [Broker Configuration](reference-configuration.md#broker) -Typically, superuser roles are used for administrators and clients but also for broker-to-broker authorization. When using [geo-replication](concepts-replication), every broker needs to be able to publish to all the other clusters' topics. +Typically, superuser roles are used for administrators and clients but also for broker-to-broker authorization. When using [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other clusters' topics. Authorization can also be enabled for the proxy the proxy configuration file (`conf/proxy.conf`). If it is enabled on the proxy, the proxy will do an additional authorization check before forwarding the request to a broker. The broker will still check the authorization of the request when it receives the forwarded request. @@ -62,7 +62,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role A Pulsar [tenant](reference-terminology.md#tenant) is typically provisioned by Pulsar [instance](reference-terminology.md#instance) administrators or by some kind of self-service portal. -Tenants are managed using the [`pulsar-admin`](reference-pulsar-admin) tool. Here's an example tenant creation command: +Tenants are managed using the [`pulsar-admin`](reference-pulsar-admin.md) tool. Here's an example tenant creation command: ```shell @@ -86,7 +86,7 @@ persistent://tenant/namespace/topic ### Managing permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication diff --git a/site2/website/versioned_docs/version-2.3.0/security-overview.md b/site2/website/versioned_docs/version-2.3.0/security-overview.md index 2f3fc035ad4cb..3f08cb29ad854 100644 --- a/site2/website/versioned_docs/version-2.3.0/security-overview.md +++ b/site2/website/versioned_docs/version-2.3.0/security-overview.md @@ -20,7 +20,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.3.0/security-tls-authentication.md b/site2/website/versioned_docs/version-2.3.0/security-tls-authentication.md index e6921bd44181f..d34ecbc1fdcf3 100644 --- a/site2/website/versioned_docs/version-2.3.0/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.3.0/security-tls-authentication.md @@ -6,9 +6,9 @@ sidebar_label: "Authentication using TLS" ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -102,7 +102,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -122,7 +122,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.3.0/security-tls-transport.md b/site2/website/versioned_docs/version-2.3.0/security-tls-transport.md index d4a36de0a3dcd..51e90aa766080 100644 --- a/site2/website/versioned_docs/version-2.3.0/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.3.0/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text, which means that all data is sent in the clear. TLS can be used to encrypt this traffic so that it cannot be snooped by a man-in-the-middle attacker. -TLS can be configured for both encryption and authentication. You may configure just TLS transport encryption, which is covered in this guide. TLS authentication is covered [elsewhere](security-tls-authentication.md). Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +TLS can be configured for both encryption and authentication. You may configure just TLS transport encryption, which is covered in this guide. TLS authentication is covered [elsewhere](security-tls-authentication.md). Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may have a performance impact due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -A third kind of key pair, **client key pairs**, are used for [client authentication](security-tls-authentication). +A third kind of key pair, **client key pairs**, are used for [client authentication](security-tls-authentication.md). The **certificate authority** private key should be stored in a very secure location (a fully encrypted, disconnected, air gapped computer). The certificate authority public key, the **trust cert**, can be freely shared. @@ -27,7 +27,7 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server they are talking to has a key pair that was signed by the certificate authority. A man-in-the-middle attacker would not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that was signed by the certificate authority. The Common Name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that was signed by the certificate authority. The Common Name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). ## Creating TLS Certificates @@ -128,7 +128,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Broker Configuration -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you'll need to make some changes to `broker.conf`, which is located in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you'll need to make some changes to `broker.conf`, which is located in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -199,7 +199,7 @@ The examples below show hostname verification being disabled for the Java client ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You'll need to add the following parameters to that file to use TLS transport with Pulsar's CLI tools: diff --git a/site2/website/versioned_docs/version-2.3.0/security-token-admin.md b/site2/website/versioned_docs/version-2.3.0/security-token-admin.md index 4accb85a75ae0..a4370fb3f8190 100644 --- a/site2/website/versioned_docs/version-2.3.0/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.3.0/security-token-admin.md @@ -162,7 +162,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.3.0/security-token-client.md b/site2/website/versioned_docs/version-2.3.0/security-token-client.md index b955bcddacf7d..2ef64e045e3b7 100644 --- a/site2/website/versioned_docs/version-2.3.0/security-token-client.md +++ b/site2/website/versioned_docs/version-2.3.0/security-token-client.md @@ -24,12 +24,12 @@ eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJKb2UifQ.ipevRNuRP6HflG8cFKnmUPtypruRC4fb1DWtoLL6 Application specifies the token when you are creating the client instance. An alternative is to pass a "token supplier" (a function that returns the token when the client library needs one). -See [Token authentication admin](security-token-admin) for a reference on how to enable token +See [Token authentication admin](security-token-admin.md) for a reference on how to enable token authentication on a Pulsar cluster. ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.3.0/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.3.0/sql-deployment-configurations.md index 639ac988b3409..069c6ac9c92da 100644 --- a/site2/website/versioned_docs/version-2.3.0/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.3.0/sql-deployment-configurations.md @@ -141,7 +141,7 @@ Since Pulsar SQL is powered by [Trino (formerly Presto SQL)](https://trino.io), :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.3.0/sql-overview.md b/site2/website/versioned_docs/version-2.3.0/sql-overview.md index 025fcd9a808ea..f5748a1b864a4 100644 --- a/site2/website/versioned_docs/version-2.3.0/sql-overview.md +++ b/site2/website/versioned_docs/version-2.3.0/sql-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Overview" original_id: sql-overview --- -One of the common use cases of Pulsar is storing streams of event data. Often the event data is structured which predefined fields. There is tremendous value for users to be able to query the existing data that is already stored in Pulsar topics. With the implementation of the [Schema Registry](concepts-schema-registry), structured data can be stored in Pulsar and allows for the potential to query that data via SQL language. +One of the common use cases of Pulsar is storing streams of event data. Often the event data is structured which predefined fields. There is tremendous value for users to be able to query the existing data that is already stored in Pulsar topics. With the implementation of the [Schema Registry](concepts-schema-registry.md), structured data can be stored in Pulsar and allows for the potential to query that data via SQL language. By leveraging [Presto](https://prestosql.io/), we have created a method for users to be able to query structured data stored within Pulsar in a very efficient and scalable manner. We will discuss why this very efficient and scalable in the [Performance](#performance) section below. diff --git a/site2/website/versioned_docs/version-2.3.0/standalone-docker.md b/site2/website/versioned_docs/version-2.3.0/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.3.0/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.3.0/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.3.0/standalone.md b/site2/website/versioned_docs/version-2.3.0/standalone.md index 6f3338a1dba77..722b4215ad3a9 100644 --- a/site2/website/versioned_docs/version-2.3.0/standalone.md +++ b/site2/website/versioned_docs/version-2.3.0/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For the purposes of local development and testing, you can run Pulsar in standalone mode on your own machine. Standalone mode includes a Pulsar broker as well as the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Run Pulsar Standalone Manually @@ -103,7 +103,7 @@ pulsar-io-aerospike-@pulsar:version@.nar > If you are running Pulsar in a bare metal cluster, you need to make sure `connectors` tarball is unzipped in every broker's pulsar directory > (or in every function-worker's pulsar directory if you are running a separate worker cluster for Pulsar functions). > -> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), > you can use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ## Installing Tiered Storage Offloaders (optional) @@ -112,7 +112,7 @@ pulsar-io-aerospike-@pulsar:version@.nar > If you would like to enable tiered storage feature, you can follow the instructions as below; otherwise you can > skip this section for now. -To get started using [tiered storage offloaders](concepts-tiered-storage), you'll need to download the offloaders tarball release on every broker node in +To get started using [tiered storage offloaders](concepts-tiered-storage.md), you'll need to download the offloaders tarball release on every broker node in one of the following ways: * by clicking the link below and downloading the release from an Apache mirror: @@ -152,7 +152,7 @@ For more details of how to configure tiered storage feature, you could reference > > If you are running Pulsar in a bare metal cluster, you need to make sure `offloaders` tarball is unzipped in every broker's pulsar directory > -> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), > you can use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. @@ -239,12 +239,12 @@ hello-pulsar ## Using Pulsar clients locally -Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: +Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: * `http://localhost:8080` * `pulsar://localhost:6650` -Here's an example producer for a Pulsar topic using the [Java](client-libraries-java) client: +Here's an example producer for a Pulsar topic using the [Java](client-libraries-java.md) client: ```java @@ -255,7 +255,7 @@ Producer producer = client.newProducer().topic("my-topic").create(); ``` -Here's an example [Python](client-libraries-python) producer: +Here's an example [Python](client-libraries-python.md) producer: ```python @@ -266,7 +266,7 @@ producer = client.create_producer('my-topic') ``` -Finally, here's an example [C++](client-libraries-cpp) producer: +Finally, here's an example [C++](client-libraries-cpp.md) producer: ```cpp diff --git a/site2/website/versioned_docs/version-2.3.1/admin-api-clusters.md b/site2/website/versioned_docs/version-2.3.1/admin-api-clusters.md index 04db2f60759bc..1255ad25d7aac 100644 --- a/site2/website/versioned_docs/version-2.3.1/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.3.1/admin-api-clusters.md @@ -88,7 +88,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -112,11 +112,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block > - For the latest and complete information about `Java admin API`, including classes, methods, descriptions, and more, see [Java admin API doc](https://pulsar.apache.org/api/admin/). -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -121,7 +121,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.3.1/client-libraries-go.md b/site2/website/versioned_docs/version-2.3.1/client-libraries-go.md index 0d7c7860eca53..9a8e2523f1c75 100644 --- a/site2/website/versioned_docs/version-2.3.1/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.3.1/client-libraries-go.md @@ -16,7 +16,7 @@ The Pulsar Go client can be used to create Pulsar [producers](#producers), [cons ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Installing go package @@ -50,7 +50,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -68,7 +68,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -223,14 +223,14 @@ Parameter | Description | Default :---------|:------------|:------- `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | | `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/) and [`ZSTD`](https://facebook.github.io/zstd/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | ## Consumers @@ -485,7 +485,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.3.1/client-libraries-java.md b/site2/website/versioned_docs/version-2.3.1/client-libraries-java.md index ecf0eba351286..9eb8555a83610 100644 --- a/site2/website/versioned_docs/version-2.3.1/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.3.1/client-libraries-java.md @@ -5,16 +5,16 @@ sidebar_label: "Java" original_id: client-libraries-java --- -The Pulsar Java client can be used both to create Java producers, consumers, and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +The Pulsar Java client can be used both to create Java producers, consumers, and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. Javadoc for the Pulsar client is divided up into two domains, by package: Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) -This document will focus only on the client API for producing and consuming messages on Pulsar topics. For a guide to using the Java admin client, see [The Pulsar admin interface](admin-api-overview). +This document will focus only on the client API for producing and consuming messages on Pulsar topics. For a guide to using the Java admin client, see [The Pulsar admin interface](admin-api-overview.md). ## Installation @@ -54,7 +54,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -72,7 +72,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -93,7 +93,7 @@ PulsarClient client = PulsarClient.builder() ``` > #### Default broker URLs for standalone clusters -> If you're running a cluster in [standalone mode](getting-started-standalone), the broker will be available at the `pulsar://localhost:6650` URL by default. +> If you're running a cluster in [standalone mode](getting-started-standalone.md), the broker will be available at the `pulsar://localhost:6650` URL by default. Check out the Javadoc for the {@inject: javadoc:PulsarClient:/client/org/apache/pulsar/client/api/PulsarClient} class for a full listing of configurable parameters. @@ -170,7 +170,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -382,7 +382,7 @@ The code sample above shows pointing the `Reader` object to a specific message ( ## Schemas -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](concepts-schema-registry) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. Here's an example: +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](concepts-schema-registry.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. Here's an example: ```java @@ -477,7 +477,7 @@ The following schema formats are currently available for Java: ## Authentication -Pulsar currently supports two authentication schemes: [TLS](security-tls-authentication.md) and [Athenz](security-athenz). The Pulsar Java client can be used with both. +Pulsar currently supports two authentication schemes: [TLS](security-tls-authentication.md) and [Athenz](security-athenz.md). The Pulsar Java client can be used with both. ### TLS Authentication @@ -504,7 +504,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` diff --git a/site2/website/versioned_docs/version-2.3.1/client-libraries-python.md b/site2/website/versioned_docs/version-2.3.1/client-libraries-python.md index 512633622b937..775c3e7eb0820 100644 --- a/site2/website/versioned_docs/version-2.3.1/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.3.1/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -The Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +The Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. ## Installation diff --git a/site2/website/versioned_docs/version-2.3.1/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.3.1/client-libraries-websocket.md index a6e60361749de..0b31ffe78161a 100644 --- a/site2/website/versioned_docs/version-2.3.1/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.3.1/client-libraries-websocket.md @@ -4,14 +4,14 @@ title: Pulsar WebSocket API sidebar_label: "WebSocket" --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.3.1/client-libraries.md b/site2/website/versioned_docs/version-2.3.1/client-libraries.md index 536cd0ccda82f..8f91dfbbbdd8c 100644 --- a/site2/website/versioned_docs/version-2.3.1/client-libraries.md +++ b/site2/website/versioned_docs/version-2.3.1/client-libraries.md @@ -8,12 +8,12 @@ Pulsar supports the following client libraries: |Language|Documentation|Release note|Code repo |---|---|---|--- -Java |- [User doc](client-libraries-java)

    - [API doc](https://pulsar.apache.org/api/client/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client) -C++ | - [User doc](client-libraries-cpp)

    - [API doc](https://pulsar.apache.org/api/cpp/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp) -Python | - [User doc](client-libraries-python)

    - [API doc](https://pulsar.apache.org/api/python/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) -WebSocket| [User doc](client-libraries-websocket) | [Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-websocket) +Java |- [User doc](client-libraries-java.md)

    - [API doc](https://pulsar.apache.org/api/client/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client) +C++ | - [User doc](client-libraries-cpp.md)

    - [API doc](https://pulsar.apache.org/api/cpp/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp) +Python | - [User doc](client-libraries-python.md)

    - [API doc](https://pulsar.apache.org/api/python/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) +WebSocket| [User doc](client-libraries-websocket.md) | [Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-websocket) Go client|[User doc](client-libraries-go.md)|[Here](https://github.com/apache/pulsar-client-go/blob/master/CHANGELOG) |[Here](https://github.com/apache/pulsar-client-go) -Node.js|[User doc](client-libraries-node)|[Here](https://github.com/apache/pulsar-client-node/releases) |[Here](https://github.com/apache/pulsar-client-node) +Node.js|[User doc](client-libraries-node.md)|[Here](https://github.com/apache/pulsar-client-node/releases) |[Here](https://github.com/apache/pulsar-client-node) C# |[User doc](client-libraries-dotnet.md)| [Here](https://github.com/apache/pulsar-dotpulsar/blob/master/CHANGELOG)|[Here](https://github.com/apache/pulsar-dotpulsar) :::note diff --git a/site2/website/versioned_docs/version-2.3.1/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.3.1/concepts-architecture-overview.md index 0d96b59f8ae91..1ce346fc16941 100644 --- a/site2/website/versioned_docs/version-2.3.1/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.3.1/concepts-architecture-overview.md @@ -4,7 +4,7 @@ title: Architecture Overview sidebar_label: "Architecture" --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -16,20 +16,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers. The producers connect to the brokers to publish messages and the consumers connect to the brokers to consume the messages. -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -39,9 +39,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -139,17 +139,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -157,7 +157,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.3.1/concepts-authentication.md b/site2/website/versioned_docs/version-2.3.1/concepts-authentication.md index 335da8dc5a519..fbefead69f382 100644 --- a/site2/website/versioned_docs/version-2.3.1/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.3.1/concepts-authentication.md @@ -4,5 +4,5 @@ title: Authentication and Authorization sidebar_label: "Authentication and Authorization" --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.3.1/concepts-clients.md b/site2/website/versioned_docs/version-2.3.1/concepts-clients.md index 65201b5ce1cd2..c0fe61cd06e76 100644 --- a/site2/website/versioned_docs/version-2.3.1/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.3.1/concepts-clients.md @@ -4,12 +4,12 @@ title: Pulsar Clients sidebar_label: "Clients" --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -37,7 +37,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.3.1/concepts-messaging.md b/site2/website/versioned_docs/version-2.3.1/concepts-messaging.md index 4cb9dbaf1c850..194bf41bd8ea9 100644 --- a/site2/website/versioned_docs/version-2.3.1/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.3.1/concepts-messaging.md @@ -24,7 +24,7 @@ Publish time | The timestamp of when the message was published (automatically ap Event time | An optional timestamp that applications can attach to the message representing when something happened, e.g. when the message was processed. The event time of a message is 0 if none is explicitly set. -> For a more in-depth breakdown of Pulsar message contents, see the documentation on Pulsar's [binary protocol](developing-binary-protocol). +> For a more in-depth breakdown of Pulsar message contents, see the documentation on Pulsar's [binary protocol](developing-binary-protocol.md). ## Producers @@ -75,7 +75,7 @@ Messages can be acknowledged either one by one or cumulatively. With cumulative ### Listeners -Client libraries can provide their own listener implementations for consumers. The [Java client](client-libraries-java), for example, provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries can provide their own listener implementations for consumers. The [Java client](client-libraries-java.md), for example, provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ## Topics @@ -200,7 +200,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -212,7 +212,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -250,7 +250,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -298,7 +298,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -321,12 +321,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, which means that you don't need to modify your Pulsar client code. Instead, you only need to make administrative changes (see the [Managing message deduplication](cookbooks-deduplication) cookbook for a guide). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, which means that you don't need to modify your Pulsar client code. Instead, you only need to make administrative changes (see the [Managing message deduplication](cookbooks-deduplication.md) cookbook for a guide). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.3.1/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.3.1/concepts-multi-tenancy.md index 8a17e72a4c8b3..26bd6a2fd183f 100644 --- a/site2/website/versioned_docs/version-2.3.1/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.3.1/concepts-multi-tenancy.md @@ -4,7 +4,7 @@ title: Multi Tenancy sidebar_label: "Multi Tenancy" --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -20,7 +20,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -28,7 +28,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.3.1/concepts-overview.md b/site2/website/versioned_docs/version-2.3.1/concepts-overview.md index 30b0a7fd10cc3..5551c57ffad90 100644 --- a/site2/website/versioned_docs/version-2.3.1/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.3.1/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.3.1/concepts-replication.md b/site2/website/versioned_docs/version-2.3.1/concepts-replication.md index d4653ca7aecd8..9cc3a7c3415bd 100644 --- a/site2/website/versioned_docs/version-2.3.1/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.3.1/concepts-replication.md @@ -8,7 +8,7 @@ Regardless of industries, when an unforeseen event occurs and brings day-to-day Pulsar's geo-replication mechanism is typically used for disaster recovery, enabling the replication of persistently stored message data across multiple data centers. For instance, your application is publishing data in one region and you would like to process it for consumption in other regions. With Pulsar’s geo-replication mechanism, messages can be produced and consumed in different geo-locations. -The diagram below illustrates the process of [geo-replication](administration-geo). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. +The diagram below illustrates the process of [geo-replication](administration-geo.md). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. ![A typical geo-replication example with full-mesh pattern](/assets/full-mesh-replication.svg) diff --git a/site2/website/versioned_docs/version-2.3.1/concepts-schema-registry.md b/site2/website/versioned_docs/version-2.3.1/concepts-schema-registry.md index 1c2ff47d3482e..a4c8ae8eb81e9 100644 --- a/site2/website/versioned_docs/version-2.3.1/concepts-schema-registry.md +++ b/site2/website/versioned_docs/version-2.3.1/concepts-schema-registry.md @@ -17,7 +17,7 @@ Both approaches are available in Pulsar, and you're free to adopt one or the oth #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ## Basic architecture @@ -39,7 +39,7 @@ Pulsar schemas are fairly simple data structures that consist of: ## Schema versions -In order to illustrate how schema versioning works, let's walk through an example. Imagine that the Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begin sending messages: +In order to illustrate how schema versioning works, let's walk through an example. Imagine that the Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begin sending messages: ```java diff --git a/site2/website/versioned_docs/version-2.3.1/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.3.1/concepts-tiered-storage.md index f0bbde61718fb..ec031253ccd8d 100644 --- a/site2/website/versioned_docs/version-2.3.1/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.3.1/concepts-tiered-storage.md @@ -14,4 +14,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.3.1/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.3.1/concepts-topic-compaction.md index 790ce76b7c5e0..300cf68956dd1 100644 --- a/site2/website/versioned_docs/version-2.3.1/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.3.1/concepts-topic-compaction.md @@ -6,7 +6,7 @@ sidebar_label: "Topic Compaction" Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -23,7 +23,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.3.1/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.3.1/cookbooks-compaction.md index 0720019aaa035..214e899fa086d 100644 --- a/site2/website/versioned_docs/version-2.3.1/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.3.1/cookbooks-compaction.md @@ -44,7 +44,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.3.1/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.3.1/cookbooks-non-persistent.md index d40c4fbd0cf01..1346d19a04263 100644 --- a/site2/website/versioned_docs/version-2.3.1/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.3.1/cookbooks-non-persistent.md @@ -40,7 +40,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -54,7 +54,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.3.1/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.3.1/cookbooks-partitioned.md index 25896934788fc..c857d019edc02 100644 --- a/site2/website/versioned_docs/version-2.3.1/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.3.1/cookbooks-partitioned.md @@ -3,4 +3,4 @@ id: cookbooks-partitioned title: Partitioned topics sidebar_label: "Partitioned Topics" --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.3.1/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.3.1/cookbooks-retention-expiry.md index 86160c3f7ecf8..cd764fad6eb37 100644 --- a/site2/website/versioned_docs/version-2.3.1/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.3.1/cookbooks-retention-expiry.md @@ -21,7 +21,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.3.1/deploy-aws.md b/site2/website/versioned_docs/version-2.3.1/deploy-aws.md index 5db846fe790f2..60ac4c22d7672 100644 --- a/site2/website/versioned_docs/version-2.3.1/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.3.1/deploy-aws.md @@ -4,7 +4,7 @@ title: Deploying a Pulsar cluster on AWS using Terraform and Ansible sidebar_label: "Amazon Web Services" --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -209,7 +209,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.3.1/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.3.1/deploy-bare-metal-multi-cluster.md index 41ce8d55e6603..4e8eae3198b9d 100644 --- a/site2/website/versioned_docs/version-2.3.1/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.3.1/deploy-bare-metal-multi-cluster.md @@ -9,17 +9,17 @@ original_id: deploy-bare-metal-multi-cluster 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you're interested in experimenting with Pulsar or using it in a startup or on a single team, we recommend opting for a single cluster. For instructions on deploying a single cluster, -see the guide [here](deploy-bare-metal). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and make sure it is installed under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and make sure it is installed under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you could reference this [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you could reference this [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar *instance* consists of multiple Pulsar clusters working in unison. Clusters can be distributed across data centers or geographical regions and can replicate amongst themselves using [geo-replication](administration-geo). Deploying a multi-cluster Pulsar instance involves the following basic steps: +A Pulsar *instance* consists of multiple Pulsar clusters working in unison. Clusters can be distributed across data centers or geographical regions and can replicate amongst themselves using [geo-replication](administration-geo.md). Deploying a multi-cluster Pulsar instance involves the following basic steps: * Deploying two separate [ZooKeeper](#deploying-zookeeper) quorums: a [local](#deploying-local-zookeeper) quorum for each cluster in the instance and a [configuration store](#configuration-store) quorum for instance-wide tasks * Initializing [cluster metadata](#cluster-metadata-initialization) for each cluster @@ -29,7 +29,7 @@ A Pulsar *instance* consists of multiple Pulsar clusters working in unison. Clus If you're deploying a single Pulsar cluster, see the [Clusters and Brokers](getting-started-standalone.md#starting-the-cluster) guide. > #### Running Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes. If you'd like to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you're looking to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes. If you'd like to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you're looking to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). ## System requirement Currently, Pulsar is available for 64-bit **macOS**, **Linux**, and **Windows**. To use Pulsar, you need to install 64-bit JRE/JDK 8 or later versions. @@ -246,7 +246,7 @@ As you can see from the example above, the following needs to be specified: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you're using [TLS](security-tls-transport), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -359,17 +359,17 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). -You can also use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can also use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > #### Service discovery already provided by many scheduling systems -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you're running Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you're running Pulsar on such a system, you may not need to provide your own service discovery mechanism. ### Service discovery setup -The service discovery mechanism included with Pulsar maintains a list of active brokers, stored in ZooKeeper, and supports lookup using HTTP and also Pulsar's [binary protocol](developing-binary-protocol). +The service discovery mechanism included with Pulsar maintains a list of active brokers, stored in ZooKeeper, and supports lookup using HTTP and also Pulsar's [binary protocol](developing-binary-protocol.md). To get started setting up Pulsar's built-in service discovery, you need to change a few parameters in the [`conf/discovery.conf`](reference-configuration.md#service-discovery) configuration file. Set the [`zookeeperServers`](reference-configuration.md#service-discovery-zookeeperServers) parameter to the cluster's ZooKeeper quorum connection string and the [`configurationStoreServers`](reference-configuration.md#service-discovery-configurationStoreServers) setting to the [configuration store](reference-terminology.md#configuration-store) quorum connection string. @@ -394,7 +394,7 @@ $ bin/pulsar-daemon start discovery ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.3.1/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.3.1/deploy-bare-metal.md index 33ad5736a2732..af5bdae2a3e18 100644 --- a/site2/website/versioned_docs/version-2.3.1/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.3.1/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you're interested in experimenting with Pulsar or using it in a startup or on a single team, we recommend opting for a single cluster. If you do need to run a multi-cluster Pulsar instance, -however, see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +however, see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and make sure it is installed under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and make sure it is installed under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you could reference this [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you could reference this [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -264,9 +264,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (we don't recommend using a different port). -`--web-service-url-tls` | If you're using [TLS](security-tls-transport), you'll also need to specify a TLS web service URL for the cluster. The default port is 8443 (we don't recommend using a different port). +`--web-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster. The default port is 8443 (we don't recommend using a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (we don't recommend using a different port). -`--broker-service-url-tls` | If you're using [TLS](security-tls-transport), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (we don't recommend using a different port). +`--broker-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (we don't recommend using a different port). ## Deploying a BookKeeper cluster @@ -376,7 +376,7 @@ webServicePortTls=8443 ### Enabling Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable function worker, by setting `functionsWorkerEnabled` to `true`. diff --git a/site2/website/versioned_docs/version-2.3.1/deploy-dcos.md b/site2/website/versioned_docs/version-2.3.1/deploy-dcos.md index 07f446edaffd6..599d3a1b0454c 100644 --- a/site2/website/versioned_docs/version-2.3.1/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.3.1/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -To enable all built-in [Pulsar IO](io-overview) connectors in your Pulsar deploymente, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). +To enable all built-in [Pulsar IO](io-overview.md) connectors in your Pulsar deploymente, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.3.1/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.3.1/deploy-kubernetes.md index 4e170dcd4711d..7455247c901d4 100644 --- a/site2/website/versioned_docs/version-2.3.1/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.3.1/deploy-kubernetes.md @@ -5,6 +5,6 @@ sidebar_label: "Kubernetes" --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.3.1/deploy-monitoring.md b/site2/website/versioned_docs/version-2.3.1/deploy-monitoring.md index adf3587b7aeb3..d581fe2edfacb 100644 --- a/site2/website/versioned_docs/version-2.3.1/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.3.1/deploy-monitoring.md @@ -112,7 +112,7 @@ http://$FUNCTIONS_WORKER_ADDRESS:$WORKER_PORT/metrics: You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy). +When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy.md). ## Dashboards @@ -120,7 +120,7 @@ When you collect time series statistics, the major problem is to make sure the n ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager). +The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.3.1/develop-schema.md b/site2/website/versioned_docs/version-2.3.1/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.3.1/develop-schema.md +++ b/site2/website/versioned_docs/version-2.3.1/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.3.1/functions-overview.md b/site2/website/versioned_docs/version-2.3.1/functions-overview.md index a0712e9f49bea..146b7492407ae 100644 --- a/site2/website/versioned_docs/version-2.3.1/functions-overview.md +++ b/site2/website/versioned_docs/version-2.3.1/functions-overview.md @@ -155,7 +155,7 @@ class RoutingFunction(Function): ## Command-line interface -Pulsar Functions are managed using the [`pulsar-admin`](reference-pulsar-admin) CLI tool (in particular the [`functions`](reference-pulsar-admin.md#functions) command). Here's an example command that would run a function in [local run mode](#local-run-mode): +Pulsar Functions are managed using the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool (in particular the [`functions`](reference-pulsar-admin.md#functions) command). Here's an example command that would run a function in [local run mode](#local-run-mode): ```bash @@ -493,7 +493,7 @@ $ bin/pulsar-admin functions create \ ## Metrics -Pulsar Functions that use the [Pulsar Functions SDK](#the-pulsar-functions-sdk) can publish metrics to Pulsar. For more information, see [Metrics for Pulsar Functions](functions-metrics). +Pulsar Functions that use the [Pulsar Functions SDK](#the-pulsar-functions-sdk) can publish metrics to Pulsar. For more information, see [Metrics for Pulsar Functions](functions-metrics.md). ## State storage diff --git a/site2/website/versioned_docs/version-2.3.1/getting-started-docker.md b/site2/website/versioned_docs/version-2.3.1/getting-started-docker.md index 6d06377e85f64..c244f33ef5de8 100644 --- a/site2/website/versioned_docs/version-2.3.1/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.3.1/getting-started-docker.md @@ -58,8 +58,8 @@ If you start Pulsar successfully, you will see `INFO`-level log messages like th ## Start publishing and consuming messages -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -114,7 +114,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.3.1/getting-started-standalone.md b/site2/website/versioned_docs/version-2.3.1/getting-started-standalone.md index fbbbfec4abad0..8d329ee32c06f 100644 --- a/site2/website/versioned_docs/version-2.3.1/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.3.1/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For the purposes of local development and testing, you can run Pulsar in standalone mode on your own machine. Standalone mode includes a Pulsar broker as well as the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Run Pulsar Standalone Manually @@ -103,7 +103,7 @@ pulsar-io-aerospike-@pulsar:version@.nar > If you are running Pulsar in a bare metal cluster, you need to make sure `connectors` tarball is unzipped in every broker's pulsar directory > (or in every function-worker's pulsar directory if you are running a separate worker cluster for Pulsar functions). > -> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), > you can use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ## Installing Tiered Storage Offloaders (optional) @@ -112,7 +112,7 @@ pulsar-io-aerospike-@pulsar:version@.nar > If you would like to enable tiered storage feature, you can follow the instructions as below; otherwise you can > skip this section for now. -To get started using [tiered storage offloaders](concepts-tiered-storage), you'll need to download the offloaders tarball release on every broker node in +To get started using [tiered storage offloaders](concepts-tiered-storage.md), you'll need to download the offloaders tarball release on every broker node in one of the following ways: * by clicking the link below and downloading the release from an Apache mirror: @@ -152,7 +152,7 @@ For more details of how to configure tiered storage feature, you could reference > > If you are running Pulsar in a bare metal cluster, you need to make sure `offloaders` tarball is unzipped in every broker's pulsar directory > -> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), > you can use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. @@ -220,12 +220,12 @@ If the message has been successfully published to the topic, you should see a co ## Using Pulsar clients locally -Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: +Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: * `http://localhost:8080` * `pulsar://localhost:6650` -Here's an example producer for a Pulsar topic using the [Java](client-libraries-java) client: +Here's an example producer for a Pulsar topic using the [Java](client-libraries-java.md) client: ```java @@ -236,7 +236,7 @@ Producer producer = client.newProducer().topic("my-topic").create(); ``` -Here's an example [Python](client-libraries-python) producer: +Here's an example [Python](client-libraries-python.md) producer: ```python @@ -247,7 +247,7 @@ producer = client.create_producer('my-topic') ``` -Finally, here's an example [C++](client-libraries-cpp) producer: +Finally, here's an example [C++](client-libraries-cpp.md) producer: ```cpp diff --git a/site2/website/versioned_docs/version-2.3.1/io-cdc.md b/site2/website/versioned_docs/version-2.3.1/io-cdc.md index df373977e3eaa..a3527dc0a9d65 100644 --- a/site2/website/versioned_docs/version-2.3.1/io-cdc.md +++ b/site2/website/versioned_docs/version-2.3.1/io-cdc.md @@ -12,8 +12,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.3.1/io-develop.md b/site2/website/versioned_docs/version-2.3.1/io-develop.md index b8b526deb2d27..cd0125d781032 100644 --- a/site2/website/versioned_docs/version-2.3.1/io-develop.md +++ b/site2/website/versioned_docs/version-2.3.1/io-develop.md @@ -8,12 +8,12 @@ original_id: io-develop This guide describes how developers can write new connectors for Pulsar IO to move data between Pulsar and other systems. It describes how to create a Pulsar IO connector. -Pulsar IO connectors are specialized [Pulsar Functions](functions-overview). So writing +Pulsar IO connectors are specialized [Pulsar Functions](functions-overview.md). So writing a Pulsar IO connector is as simple as writing a Pulsar function. Pulsar IO connectors come in two flavors: {@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}, which import data from another system, and {@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}, -which export data to another system. For example, [KinesisSink](io-kinesis) would export -the messages of a Pulsar topic to a Kinesis stream, and [RabbitmqSource](io-rabbitmq) would import +which export data to another system. For example, [KinesisSink](io-kinesis.md) would export +the messages of a Pulsar topic to a Kinesis stream, and [RabbitmqSource](io-rabbitmq.md) would import the messages of a RabbitMQ queue to a Pulsar topic. ### Developing @@ -137,7 +137,7 @@ separate integration tests to verify end-to-end functionality. In Pulsar, we are ### Packaging Once you've developed and tested your connector, you must package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. There are two approaches described +to a [Pulsar Functions](functions-overview.md) cluster. There are two approaches described here work with Pulsar Functions' runtime. If you plan to package and distribute your connector for others to use, you are obligated to @@ -153,7 +153,7 @@ The easiest approach to packaging a Pulsar IO connector is to create a NAR packa NAR stands for NiFi Archive. It is a custom packaging mechanism used by Apache NiFi, to provide a bit of Java ClassLoader isolation. For more details, you can read this [blog post](https://medium.com/hashmapinc/nifi-nar-files-explained-14113f7796fd) to understand -how NAR works. Pulsar uses the same mechanism for packaging all the [builtin connectors](io-connectors). +how NAR works. Pulsar uses the same mechanism for packaging all the [builtin connectors](io-connectors.md). All what you need is to include this [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin) in your maven project for your connector. For example: diff --git a/site2/website/versioned_docs/version-2.3.1/io-managing.md b/site2/website/versioned_docs/version-2.3.1/io-managing.md index cce795bbae86a..8c834410a5d0f 100644 --- a/site2/website/versioned_docs/version-2.3.1/io-managing.md +++ b/site2/website/versioned_docs/version-2.3.1/io-managing.md @@ -50,7 +50,7 @@ For details, consult the documentation for [individual connectors](io-overview.m ## Running Connectors -Pulsar connectors can be managed using the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands of the [`pulsar-admin`](reference-pulsar-admin) CLI tool. +Pulsar connectors can be managed using the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands of the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. ### Running sources @@ -166,8 +166,8 @@ Here's an example to submit a Cassandra sink: ## Monitoring Connectors -Since Pulsar IO connectors are running as [Pulsar Functions](functions-overview), so you can use [`functions`](reference-pulsar-admin.md#source) commands -available in the [`pulsar-admin`](reference-pulsar-admin) CLI tool. +Since Pulsar IO connectors are running as [Pulsar Functions](functions-overview.md), so you can use [`functions`](reference-pulsar-admin.md#source) commands +available in the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. ### Retrieve Connector Metadata diff --git a/site2/website/versioned_docs/version-2.3.1/io-overview.md b/site2/website/versioned_docs/version-2.3.1/io-overview.md index 2a6ecaee5883f..ea49e4b042e8c 100644 --- a/site2/website/versioned_docs/version-2.3.1/io-overview.md +++ b/site2/website/versioned_docs/version-2.3.1/io-overview.md @@ -157,7 +157,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.3.1/io-quickstart.md b/site2/website/versioned_docs/version-2.3.1/io-quickstart.md index c4230323e66f4..83a57170dfc91 100644 --- a/site2/website/versioned_docs/version-2.3.1/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.3.1/io-quickstart.md @@ -6,14 +6,14 @@ original_id: io-quickstart --- This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O in tandem with running the steps in this guide +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O in tandem with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you will be able to: - Connect your Pulsar cluster with your Cassandra cluster :::tip -1. These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However all +1. These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However all the commands used in this tutorial should be able to be used in a multi-nodes Pulsar cluster without any changes. 2. All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -260,11 +260,11 @@ configs: ``` -To learn more about Cassandra Connector, see [Cassandra Connector](io-cassandra). +To learn more about Cassandra Connector, see [Cassandra Connector](io-cassandra.md). ### Submit a Cassandra Sink -Pulsar provides the [CLI](reference-cli-tools) for running and managing Pulsar I/O connectors. +Pulsar provides the [CLI](reference-cli-tools.md) for running and managing Pulsar I/O connectors. We can run following command to sink a sink connector with type `cassandra` and config file `examples/cassandra-sink.yml`. diff --git a/site2/website/versioned_docs/version-2.3.1/pulsar-2.0.md b/site2/website/versioned_docs/version-2.3.1/pulsar-2.0.md index 560c8c1bd7cb7..07fba365e9ff2 100644 --- a/site2/website/versioned_docs/version-2.3.1/pulsar-2.0.md +++ b/site2/website/versioned_docs/version-2.3.1/pulsar-2.0.md @@ -4,13 +4,13 @@ title: Pulsar 2.0 sidebar_label: "Pulsar 2.0" --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.3.1/reference-cli-tools.md b/site2/website/versioned_docs/version-2.3.1/reference-cli-tools.md index 902f47bb408c8..a781f2f2b6a1f 100644 --- a/site2/website/versioned_docs/version-2.3.1/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.3.1/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.3.1/security-athenz.md b/site2/website/versioned_docs/version-2.3.1/security-athenz.md index ba27ba43eb9d8..6d338fc4013bd 100644 --- a/site2/website/versioned_docs/version-2.3.1/security-athenz.md +++ b/site2/website/versioned_docs/version-2.3.1/security-athenz.md @@ -75,7 +75,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.3.1/security-authorization.md b/site2/website/versioned_docs/version-2.3.1/security-authorization.md index d9d33665d066b..c5e3c04b03524 100644 --- a/site2/website/versioned_docs/version-2.3.1/security-authorization.md +++ b/site2/website/versioned_docs/version-2.3.1/security-authorization.md @@ -26,7 +26,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -58,7 +58,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -86,7 +86,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication @@ -116,7 +116,7 @@ PulsarAdmin admin = PulsarAdmin.builder() When a client is identified with multiple roles in a token (the type of role claim in the token is an array) during the authentication process, Pulsar supports to check the permissions of all the roles and further authorize the client as long as one of its roles has the required permissions. > **Note**
    -> This authorization method is only compatible with [JWT authentication](security-jwt). +> This authorization method is only compatible with [JWT authentication](security-jwt.md). To enable this authorization method, configure the authorization provider as `MultiRolesTokenAuthorizationProvider` in the `conf/broker.conf` file. diff --git a/site2/website/versioned_docs/version-2.3.1/security-overview.md b/site2/website/versioned_docs/version-2.3.1/security-overview.md index 2f3fc035ad4cb..3f08cb29ad854 100644 --- a/site2/website/versioned_docs/version-2.3.1/security-overview.md +++ b/site2/website/versioned_docs/version-2.3.1/security-overview.md @@ -20,7 +20,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.3.1/security-tls-authentication.md b/site2/website/versioned_docs/version-2.3.1/security-tls-authentication.md index e6921bd44181f..d34ecbc1fdcf3 100644 --- a/site2/website/versioned_docs/version-2.3.1/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.3.1/security-tls-authentication.md @@ -6,9 +6,9 @@ sidebar_label: "Authentication using TLS" ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -102,7 +102,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -122,7 +122,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.3.1/security-tls-transport.md b/site2/website/versioned_docs/version-2.3.1/security-tls-transport.md index d4a36de0a3dcd..51e90aa766080 100644 --- a/site2/website/versioned_docs/version-2.3.1/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.3.1/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text, which means that all data is sent in the clear. TLS can be used to encrypt this traffic so that it cannot be snooped by a man-in-the-middle attacker. -TLS can be configured for both encryption and authentication. You may configure just TLS transport encryption, which is covered in this guide. TLS authentication is covered [elsewhere](security-tls-authentication.md). Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +TLS can be configured for both encryption and authentication. You may configure just TLS transport encryption, which is covered in this guide. TLS authentication is covered [elsewhere](security-tls-authentication.md). Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may have a performance impact due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -A third kind of key pair, **client key pairs**, are used for [client authentication](security-tls-authentication). +A third kind of key pair, **client key pairs**, are used for [client authentication](security-tls-authentication.md). The **certificate authority** private key should be stored in a very secure location (a fully encrypted, disconnected, air gapped computer). The certificate authority public key, the **trust cert**, can be freely shared. @@ -27,7 +27,7 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server they are talking to has a key pair that was signed by the certificate authority. A man-in-the-middle attacker would not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that was signed by the certificate authority. The Common Name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that was signed by the certificate authority. The Common Name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). ## Creating TLS Certificates @@ -128,7 +128,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Broker Configuration -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you'll need to make some changes to `broker.conf`, which is located in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you'll need to make some changes to `broker.conf`, which is located in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -199,7 +199,7 @@ The examples below show hostname verification being disabled for the Java client ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You'll need to add the following parameters to that file to use TLS transport with Pulsar's CLI tools: diff --git a/site2/website/versioned_docs/version-2.3.1/security-token-admin.md b/site2/website/versioned_docs/version-2.3.1/security-token-admin.md index c0aabd249f4b0..e89fa71766b02 100644 --- a/site2/website/versioned_docs/version-2.3.1/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.3.1/security-token-admin.md @@ -172,7 +172,7 @@ brokerClientAuthenticationParameters=token:eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ0ZXN0 To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.3.1/security-token-client.md b/site2/website/versioned_docs/version-2.3.1/security-token-client.md index b955bcddacf7d..2ef64e045e3b7 100644 --- a/site2/website/versioned_docs/version-2.3.1/security-token-client.md +++ b/site2/website/versioned_docs/version-2.3.1/security-token-client.md @@ -24,12 +24,12 @@ eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJKb2UifQ.ipevRNuRP6HflG8cFKnmUPtypruRC4fb1DWtoLL6 Application specifies the token when you are creating the client instance. An alternative is to pass a "token supplier" (a function that returns the token when the client library needs one). -See [Token authentication admin](security-token-admin) for a reference on how to enable token +See [Token authentication admin](security-token-admin.md) for a reference on how to enable token authentication on a Pulsar cluster. ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.3.1/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.3.1/sql-deployment-configurations.md index 639ac988b3409..069c6ac9c92da 100644 --- a/site2/website/versioned_docs/version-2.3.1/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.3.1/sql-deployment-configurations.md @@ -141,7 +141,7 @@ Since Pulsar SQL is powered by [Trino (formerly Presto SQL)](https://trino.io), :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.3.1/sql-overview.md b/site2/website/versioned_docs/version-2.3.1/sql-overview.md index 2f827f4267933..5b01dfde56e7f 100644 --- a/site2/website/versioned_docs/version-2.3.1/sql-overview.md +++ b/site2/website/versioned_docs/version-2.3.1/sql-overview.md @@ -4,7 +4,7 @@ title: Pulsar SQL Overview sidebar_label: "Overview" --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL)](https://trino.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL.md)](https://trino.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.3.1/standalone-docker.md b/site2/website/versioned_docs/version-2.3.1/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.3.1/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.3.1/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.3.1/standalone.md b/site2/website/versioned_docs/version-2.3.1/standalone.md index a5d7c47908105..7eec2156eeca7 100644 --- a/site2/website/versioned_docs/version-2.3.1/standalone.md +++ b/site2/website/versioned_docs/version-2.3.1/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For the purposes of local development and testing, you can run Pulsar in standalone mode on your own machine. Standalone mode includes a Pulsar broker as well as the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Run Pulsar Standalone Manually @@ -103,7 +103,7 @@ pulsar-io-aerospike-@pulsar:version@.nar > If you are running Pulsar in a bare metal cluster, you need to make sure `connectors` tarball is unzipped in every broker's pulsar directory > (or in every function-worker's pulsar directory if you are running a separate worker cluster for Pulsar functions). > -> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), > you can use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ## Installing Tiered Storage Offloaders (optional) @@ -112,7 +112,7 @@ pulsar-io-aerospike-@pulsar:version@.nar > If you would like to enable tiered storage feature, you can follow the instructions as below; otherwise you can > skip this section for now. -To get started using [tiered storage offloaders](concepts-tiered-storage), you'll need to download the offloaders tarball release on every broker node in +To get started using [tiered storage offloaders](concepts-tiered-storage.md), you'll need to download the offloaders tarball release on every broker node in one of the following ways: * by clicking the link below and downloading the release from an Apache mirror: @@ -152,7 +152,7 @@ For more details of how to configure tiered storage feature, you could reference > > If you are running Pulsar in a bare metal cluster, you need to make sure `offloaders` tarball is unzipped in every broker's pulsar directory > -> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +> If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), > you can use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. @@ -220,12 +220,12 @@ If the message has been successfully published to the topic, you should see a co ## Using Pulsar clients locally -Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: +Pulsar currently offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs for interacting with your cluster: * `http://localhost:8080` * `pulsar://localhost:6650` -Here's an example producer for a Pulsar topic using the [Java](client-libraries-java) client: +Here's an example producer for a Pulsar topic using the [Java](client-libraries-java.md) client: ```java @@ -236,7 +236,7 @@ Producer producer = client.newProducer().topic("my-topic").create(); ``` -Here's an example [Python](client-libraries-python) producer: +Here's an example [Python](client-libraries-python.md) producer: ```python @@ -247,7 +247,7 @@ producer = client.create_producer('my-topic') ``` -Finally, here's an example [C++](client-libraries-cpp) producer: +Finally, here's an example [C++](client-libraries-cpp.md) producer: ```cpp diff --git a/site2/website/versioned_docs/version-2.3.2/admin-api-clusters.md b/site2/website/versioned_docs/version-2.3.2/admin-api-clusters.md index 04db2f60759bc..1255ad25d7aac 100644 --- a/site2/website/versioned_docs/version-2.3.2/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.3.2/admin-api-clusters.md @@ -88,7 +88,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -112,11 +112,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block > - For the latest and complete information about `Java admin API`, including classes, methods, descriptions, and more, see [Java admin API doc](https://pulsar.apache.org/api/admin/). -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -121,7 +121,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.3.2/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.3.2/client-libraries-cpp.md index 5ba9e03d89521..b709463ff0309 100644 --- a/site2/website/versioned_docs/version-2.3.2/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.3.2/client-libraries-cpp.md @@ -661,7 +661,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema This section describes some examples about schema. For more information about -schema, see [Pulsar schema](schema-get-started). +schema, see [Pulsar schema](schema-get-started.md). ### Avro schema diff --git a/site2/website/versioned_docs/version-2.3.2/client-libraries-go.md b/site2/website/versioned_docs/version-2.3.2/client-libraries-go.md index 33059163f4c6b..d3c2fa5d2fc85 100644 --- a/site2/website/versioned_docs/version-2.3.2/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.3.2/client-libraries-go.md @@ -55,7 +55,7 @@ $ ./test_example ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -81,7 +81,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -1022,7 +1022,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker @@ -1042,7 +1042,7 @@ opts := pulsar.ClientOptions{ ## OAuth2 authentication -To use [OAuth2 authentication](security-oauth2), you'll need to configure your client to perform the following operations. +To use [OAuth2 authentication](security-oauth2.md), you'll need to configure your client to perform the following operations. This example shows how to configure OAuth2 authentication. ```go diff --git a/site2/website/versioned_docs/version-2.3.2/client-libraries-java.md b/site2/website/versioned_docs/version-2.3.2/client-libraries-java.md index 2c349dbf12f12..1058b0896fa78 100644 --- a/site2/website/versioned_docs/version-2.3.2/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.3.2/client-libraries-java.md @@ -10,7 +10,7 @@ import TabItem from '@theme/TabItem'; ```` -You can use a Pulsar Java client to create the Java [producer](#producer), [consumer](#consumer), [reader](#reader) and [TableView](#tableview) of messages and to perform [administrative tasks](admin-api-overview). The current Java client version is **@pulsar:version@**. +You can use a Pulsar Java client to create the Java [producer](#producer), [consumer](#consumer), [reader](#reader) and [TableView](#tableview) of messages and to perform [administrative tasks](admin-api-overview.md). The current Java client version is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), [reader](#reader) and [TableView](#tableview) of a Java client are thread-safe. @@ -19,10 +19,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | [The producer and consumer API](https://pulsar.apache.org/api/client/) | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Include both `pulsar-client` and `pulsar-client-admin`
    Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.
    In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -69,7 +69,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -95,7 +95,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -126,7 +126,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -279,7 +279,7 @@ Currently, cluster-level failover can perform probes to prevent data loss, but i > #### What are the relationships between cluster-level failover and geo-replication? -The cluster-level failover is an extension of [geo-replication](concepts-replication) to improve stability and robustness. The cluster-level failover depends on geo-replication, and they have some **differences** as below. +The cluster-level failover is an extension of [geo-replication](concepts-replication.md) to improve stability and robustness. The cluster-level failover depends on geo-replication, and they have some **differences** as below. Influence |Cluster-level failover|Geo-replication |---|---|--- @@ -576,7 +576,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics cookbook](cookbooks-partitioned). +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics cookbook](cookbooks-partitioned.md). ### Async send @@ -1370,7 +1370,7 @@ tv.forEach((key, value) -> /*operations on all existing messages*/) ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producer) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producer) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -1488,7 +1488,7 @@ For example of ProtobufNativeSchema, see [`SchemaDefinition` in `Complex type`]( ## Authentication -Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2). You can use the Pulsar Java client with all of them. +Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2.md). You can use the Pulsar Java client with all of them. ### TLS Authentication @@ -1515,7 +1515,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` @@ -1552,7 +1552,7 @@ PulsarClient client = PulsarClient.builder() ### Oauth2 -The following example shows how to use [Oauth2](security-oauth2) as an authentication provider for the Pulsar Java client. +The following example shows how to use [Oauth2](security-oauth2.md) as an authentication provider for the Pulsar Java client. You can use the factory method to configure authentication for Pulsar Java client. diff --git a/site2/website/versioned_docs/version-2.3.2/client-libraries-python.md b/site2/website/versioned_docs/version-2.3.2/client-libraries-python.md index 4b6b4f9e41d6c..7ed649e86096a 100644 --- a/site2/website/versioned_docs/version-2.3.2/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.3.2/client-libraries-python.md @@ -10,7 +10,7 @@ import TabItem from '@theme/TabItem'; ```` -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [Python directory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [Python directory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. @@ -192,7 +192,7 @@ client.close() ``` ### Create a Python client with multiple advertised listeners -To ensure clients in both internal and external networks can connect to a Pulsar cluster, Pulsar introduces [advertisedListeners](concepts-multiple-advertised-listeners). +To ensure clients in both internal and external networks can connect to a Pulsar cluster, Pulsar introduces [advertisedListeners](concepts-multiple-advertised-listeners.md). The following example creates a Python client using multiple advertised listeners: diff --git a/site2/website/versioned_docs/version-2.3.2/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.3.2/client-libraries-websocket.md index a6e60361749de..0b31ffe78161a 100644 --- a/site2/website/versioned_docs/version-2.3.2/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.3.2/client-libraries-websocket.md @@ -4,14 +4,14 @@ title: Pulsar WebSocket API sidebar_label: "WebSocket" --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.3.2/client-libraries.md b/site2/website/versioned_docs/version-2.3.2/client-libraries.md index aaf4becd3d959..b6ca97de9cad9 100644 --- a/site2/website/versioned_docs/version-2.3.2/client-libraries.md +++ b/site2/website/versioned_docs/version-2.3.2/client-libraries.md @@ -14,30 +14,30 @@ Pulsar supports the following client libraries: ## Java client -For instructions on how to use the Pulsar Java client to produce and consume messages, see [Pulsar Java client](client-libraries-java). +For instructions on how to use the Pulsar Java client to produce and consume messages, see [Pulsar Java client](client-libraries-java.md). Two independent sets of Javadoc API docs are available. Library | Purpose :-------|:------- -[`org.apache.pulsar.client.api`](/api/client) | The [Pulsar Java client](client-libraries-java) is used to produce and consume messages on Pulsar topics. -[`org.apache.pulsar.client.admin`](/api/admin) | The Java client for the [Pulsar admin interface](admin-api-overview). +[`org.apache.pulsar.client.api`](/api/client) | The [Pulsar Java client](client-libraries-java.md) is used to produce and consume messages on Pulsar topics. +[`org.apache.pulsar.client.admin`](/api/admin) | The Java client for the [Pulsar admin interface](admin-api-overview.md). ## Go client -For a tutorial on using the Pulsar Go client, see [Pulsar Go client](client-libraries-go). +For a tutorial on using the Pulsar Go client, see [Pulsar Go client](client-libraries-go.md). ## Python client -For a tutorial on using the Pulsar Python client, see [Pulsar Python client](client-libraries-python). +For a tutorial on using the Pulsar Python client, see [Pulsar Python client](client-libraries-python.md). There are also [pdoc](https://github.com/BurntSushi/pdoc)-generated API docs for the Python client [here](/api/python). ## C++ client -For a tutorial on using the Pulsar C++ clent, see [Pulsar C++ client](client-libraries-cpp). +For a tutorial on using the Pulsar C++ clent, see [Pulsar C++ client](client-libraries-cpp.md). There are also [Doxygen](http://www.stack.nl/~dimitri/doxygen/)-generated API docs for the C++ client [here](/api/cpp). diff --git a/site2/website/versioned_docs/version-2.3.2/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.3.2/concepts-architecture-overview.md index 0d96b59f8ae91..1ce346fc16941 100644 --- a/site2/website/versioned_docs/version-2.3.2/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.3.2/concepts-architecture-overview.md @@ -4,7 +4,7 @@ title: Architecture Overview sidebar_label: "Architecture" --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -16,20 +16,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers. The producers connect to the brokers to publish messages and the consumers connect to the brokers to consume the messages. -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -39,9 +39,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -139,17 +139,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -157,7 +157,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.3.2/concepts-authentication.md b/site2/website/versioned_docs/version-2.3.2/concepts-authentication.md index 335da8dc5a519..fbefead69f382 100644 --- a/site2/website/versioned_docs/version-2.3.2/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.3.2/concepts-authentication.md @@ -4,5 +4,5 @@ title: Authentication and Authorization sidebar_label: "Authentication and Authorization" --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.3.2/concepts-clients.md b/site2/website/versioned_docs/version-2.3.2/concepts-clients.md index 17d2a8d672dd5..d986f3058dffe 100644 --- a/site2/website/versioned_docs/version-2.3.2/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.3.2/concepts-clients.md @@ -5,7 +5,7 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. diff --git a/site2/website/versioned_docs/version-2.3.2/concepts-messaging.md b/site2/website/versioned_docs/version-2.3.2/concepts-messaging.md index cae451bb2b82f..f035ebf72d48b 100644 --- a/site2/website/versioned_docs/version-2.3.2/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.3.2/concepts-messaging.md @@ -24,7 +24,7 @@ Publish time | The timestamp of when the message was published (automatically ap Event time | An optional timestamp that applications can attach to the message representing when something happened, e.g. when the message was processed. The event time of a message is 0 if none is explicitly set. -> For a more in-depth breakdown of Pulsar message contents, see the documentation on Pulsar's [binary protocol](developing-binary-protocol). +> For a more in-depth breakdown of Pulsar message contents, see the documentation on Pulsar's [binary protocol](developing-binary-protocol.md). ## Producers @@ -67,7 +67,7 @@ Messages can be received from [brokers](reference-terminology.md#broker) either ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -263,7 +263,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -275,7 +275,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -313,7 +313,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -361,7 +361,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -384,12 +384,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, which means that you don't need to modify your Pulsar client code. Instead, you only need to make administrative changes (see the [Managing message deduplication](cookbooks-deduplication) cookbook for a guide). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, which means that you don't need to modify your Pulsar client code. Instead, you only need to make administrative changes (see the [Managing message deduplication](cookbooks-deduplication.md) cookbook for a guide). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.3.2/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.3.2/concepts-multi-tenancy.md index 8a17e72a4c8b3..26bd6a2fd183f 100644 --- a/site2/website/versioned_docs/version-2.3.2/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.3.2/concepts-multi-tenancy.md @@ -4,7 +4,7 @@ title: Multi Tenancy sidebar_label: "Multi Tenancy" --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -20,7 +20,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -28,7 +28,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.3.2/concepts-overview.md b/site2/website/versioned_docs/version-2.3.2/concepts-overview.md index 119d46508ed28..bf4993a3110b5 100644 --- a/site2/website/versioned_docs/version-2.3.2/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.3.2/concepts-overview.md @@ -8,15 +8,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.3.2/concepts-replication.md b/site2/website/versioned_docs/version-2.3.2/concepts-replication.md index d4653ca7aecd8..9cc3a7c3415bd 100644 --- a/site2/website/versioned_docs/version-2.3.2/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.3.2/concepts-replication.md @@ -8,7 +8,7 @@ Regardless of industries, when an unforeseen event occurs and brings day-to-day Pulsar's geo-replication mechanism is typically used for disaster recovery, enabling the replication of persistently stored message data across multiple data centers. For instance, your application is publishing data in one region and you would like to process it for consumption in other regions. With Pulsar’s geo-replication mechanism, messages can be produced and consumed in different geo-locations. -The diagram below illustrates the process of [geo-replication](administration-geo). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. +The diagram below illustrates the process of [geo-replication](administration-geo.md). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. ![A typical geo-replication example with full-mesh pattern](/assets/full-mesh-replication.svg) diff --git a/site2/website/versioned_docs/version-2.3.2/concepts-schema-registry.md b/site2/website/versioned_docs/version-2.3.2/concepts-schema-registry.md index c2a42c1bc3f5e..7ad8eabb3fe76 100644 --- a/site2/website/versioned_docs/version-2.3.2/concepts-schema-registry.md +++ b/site2/website/versioned_docs/version-2.3.2/concepts-schema-registry.md @@ -17,7 +17,7 @@ Both approaches are available in Pulsar, and you're free to adopt one or the oth #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-go.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-go.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ## Basic architecture @@ -39,7 +39,7 @@ Pulsar schemas are fairly simple data structures that consist of: ## Schema versions -In order to illustrate how schema versioning works, let's walk through an example. Imagine that the Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begin sending messages: +In order to illustrate how schema versioning works, let's walk through an example. Imagine that the Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begin sending messages: ```java diff --git a/site2/website/versioned_docs/version-2.3.2/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.3.2/concepts-tiered-storage.md index f0bbde61718fb..ec031253ccd8d 100644 --- a/site2/website/versioned_docs/version-2.3.2/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.3.2/concepts-tiered-storage.md @@ -14,4 +14,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.3.2/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.3.2/concepts-topic-compaction.md index 790ce76b7c5e0..300cf68956dd1 100644 --- a/site2/website/versioned_docs/version-2.3.2/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.3.2/concepts-topic-compaction.md @@ -6,7 +6,7 @@ sidebar_label: "Topic Compaction" Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -23,7 +23,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.3.2/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.3.2/cookbooks-compaction.md index 0720019aaa035..214e899fa086d 100644 --- a/site2/website/versioned_docs/version-2.3.2/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.3.2/cookbooks-compaction.md @@ -44,7 +44,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.3.2/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.3.2/cookbooks-non-persistent.md index d40c4fbd0cf01..1346d19a04263 100644 --- a/site2/website/versioned_docs/version-2.3.2/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.3.2/cookbooks-non-persistent.md @@ -40,7 +40,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -54,7 +54,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.3.2/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.3.2/cookbooks-partitioned.md index 25896934788fc..c857d019edc02 100644 --- a/site2/website/versioned_docs/version-2.3.2/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.3.2/cookbooks-partitioned.md @@ -3,4 +3,4 @@ id: cookbooks-partitioned title: Partitioned topics sidebar_label: "Partitioned Topics" --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.3.2/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.3.2/cookbooks-retention-expiry.md index 86160c3f7ecf8..cd764fad6eb37 100644 --- a/site2/website/versioned_docs/version-2.3.2/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.3.2/cookbooks-retention-expiry.md @@ -21,7 +21,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.3.2/deploy-aws.md b/site2/website/versioned_docs/version-2.3.2/deploy-aws.md index 5db846fe790f2..60ac4c22d7672 100644 --- a/site2/website/versioned_docs/version-2.3.2/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.3.2/deploy-aws.md @@ -4,7 +4,7 @@ title: Deploying a Pulsar cluster on AWS using Terraform and Ansible sidebar_label: "Amazon Web Services" --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -209,7 +209,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.3.2/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.3.2/deploy-bare-metal-multi-cluster.md index 70dff45080b03..8a765d8843f43 100644 --- a/site2/website/versioned_docs/version-2.3.2/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.3.2/deploy-bare-metal-multi-cluster.md @@ -6,13 +6,13 @@ sidebar_label: "Bare metal multi-cluster" :::tip -1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster). -2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster.md). +2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo).Deploying a multi-cluster Pulsar instance consists of the following steps: +A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md).Deploying a multi-cluster Pulsar instance consists of the following steps: 1. Deploying two separate ZooKeeper quorums: a local quorum for each cluster in the instance and a configuration store quorum for instance-wide tasks 2. Initializing cluster metadata for each cluster @@ -21,7 +21,7 @@ A Pulsar instance consists of multiple Pulsar clusters working in unison. You ca > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. ## System requirement @@ -65,7 +65,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -244,7 +244,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -363,16 +363,16 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. -You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > **Service discovery already provided by many scheduling systems** -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.3.2/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.3.2/deploy-bare-metal.md index 6669476734558..614815ef1f869 100644 --- a/site2/website/versioned_docs/version-2.3.2/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.3.2/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you're interested in experimenting with Pulsar or using it in a startup or on a single team, we recommend opting for a single cluster. If you do need to run a multi-cluster Pulsar instance, -however, see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +however, see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and make sure it is installed under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and make sure it is installed under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you could reference this [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you could reference this [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -264,9 +264,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (we don't recommend using a different port). -`--web-service-url-tls` | If you're using [TLS](security-tls-transport), you'll also need to specify a TLS web service URL for the cluster. The default port is 8443 (we don't recommend using a different port). +`--web-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster. The default port is 8443 (we don't recommend using a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (we don't recommend using a different port). -`--broker-service-url-tls` | If you're using [TLS](security-tls-transport), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (we don't recommend using a different port). +`--broker-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (we don't recommend using a different port). ## Deploying a BookKeeper cluster @@ -376,7 +376,7 @@ webServicePortTls=8443 ### Enabling Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -394,7 +394,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you would like to learn more options about deploying functions worker, please checkout [Deploy and manage functions worker](functions-worker). +If you would like to learn more options about deploying functions worker, please checkout [Deploy and manage functions worker](functions-worker.md). ### Starting Brokers diff --git a/site2/website/versioned_docs/version-2.3.2/deploy-dcos.md b/site2/website/versioned_docs/version-2.3.2/deploy-dcos.md index 07f446edaffd6..599d3a1b0454c 100644 --- a/site2/website/versioned_docs/version-2.3.2/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.3.2/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -To enable all built-in [Pulsar IO](io-overview) connectors in your Pulsar deploymente, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). +To enable all built-in [Pulsar IO](io-overview.md) connectors in your Pulsar deploymente, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.3.2/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.3.2/deploy-kubernetes.md index 4e170dcd4711d..7455247c901d4 100644 --- a/site2/website/versioned_docs/version-2.3.2/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.3.2/deploy-kubernetes.md @@ -5,6 +5,6 @@ sidebar_label: "Kubernetes" --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.3.2/deploy-monitoring.md b/site2/website/versioned_docs/version-2.3.2/deploy-monitoring.md index adf3587b7aeb3..d581fe2edfacb 100644 --- a/site2/website/versioned_docs/version-2.3.2/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.3.2/deploy-monitoring.md @@ -112,7 +112,7 @@ http://$FUNCTIONS_WORKER_ADDRESS:$WORKER_PORT/metrics: You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy). +When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy.md). ## Dashboards @@ -120,7 +120,7 @@ When you collect time series statistics, the major problem is to make sure the n ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager). +The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.3.2/develop-schema.md b/site2/website/versioned_docs/version-2.3.2/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.3.2/develop-schema.md +++ b/site2/website/versioned_docs/version-2.3.2/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.3.2/functions-overview.md b/site2/website/versioned_docs/version-2.3.2/functions-overview.md index c84be36c58882..3a59d43923e09 100644 --- a/site2/website/versioned_docs/version-2.3.2/functions-overview.md +++ b/site2/website/versioned_docs/version-2.3.2/functions-overview.md @@ -163,7 +163,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.3.2/functions-worker.md b/site2/website/versioned_docs/version-2.3.2/functions-worker.md index 101ce6fe04f4f..b2f1e10aea471 100644 --- a/site2/website/versioned_docs/version-2.3.2/functions-worker.md +++ b/site2/website/versioned_docs/version-2.3.2/functions-worker.md @@ -122,7 +122,7 @@ tlsTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). **Enable Authentication Provider** @@ -149,7 +149,7 @@ properties: ``` For *Token Authentication* prodivder, add necessary settings under `properties` if needed. -See [Token Authentication](security-token-admin) for more details. +See [Token Authentication](security-token-admin.md) for more details. ``` diff --git a/site2/website/versioned_docs/version-2.3.2/getting-started-docker.md b/site2/website/versioned_docs/version-2.3.2/getting-started-docker.md index 694a93c73414d..c99dac6cc4033 100644 --- a/site2/website/versioned_docs/version-2.3.2/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.3.2/getting-started-docker.md @@ -62,8 +62,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -122,7 +122,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.3.2/getting-started-standalone.md b/site2/website/versioned_docs/version-2.3.2/getting-started-standalone.md index 1994800630b9a..dcad07a2fc853 100644 --- a/site2/website/versioned_docs/version-2.3.2/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.3.2/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -49,9 +49,9 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](reference-pulsar-admin). +`bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](reference-pulsar-admin.md). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -60,7 +60,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. #### Install other optional components @@ -111,7 +111,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -125,7 +125,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -158,12 +158,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: diff --git a/site2/website/versioned_docs/version-2.3.2/io-cdc.md b/site2/website/versioned_docs/version-2.3.2/io-cdc.md index df373977e3eaa..a3527dc0a9d65 100644 --- a/site2/website/versioned_docs/version-2.3.2/io-cdc.md +++ b/site2/website/versioned_docs/version-2.3.2/io-cdc.md @@ -12,8 +12,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.3.2/io-develop.md b/site2/website/versioned_docs/version-2.3.2/io-develop.md index 69c9a2f42c428..d361a302b2529 100644 --- a/site2/website/versioned_docs/version-2.3.2/io-develop.md +++ b/site2/website/versioned_docs/version-2.3.2/io-develop.md @@ -13,15 +13,15 @@ import TabItem from '@theme/TabItem'; This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -274,7 +274,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -305,7 +305,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). @@ -380,7 +380,7 @@ Pulsar connectors enable you to move data in and out of Pulsar easily. It is imp - Check the metrics provided by Pulsar. - Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. - Set and check your customized metrics. diff --git a/site2/website/versioned_docs/version-2.3.2/io-managing.md b/site2/website/versioned_docs/version-2.3.2/io-managing.md index cce795bbae86a..8c834410a5d0f 100644 --- a/site2/website/versioned_docs/version-2.3.2/io-managing.md +++ b/site2/website/versioned_docs/version-2.3.2/io-managing.md @@ -50,7 +50,7 @@ For details, consult the documentation for [individual connectors](io-overview.m ## Running Connectors -Pulsar connectors can be managed using the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands of the [`pulsar-admin`](reference-pulsar-admin) CLI tool. +Pulsar connectors can be managed using the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands of the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. ### Running sources @@ -166,8 +166,8 @@ Here's an example to submit a Cassandra sink: ## Monitoring Connectors -Since Pulsar IO connectors are running as [Pulsar Functions](functions-overview), so you can use [`functions`](reference-pulsar-admin.md#source) commands -available in the [`pulsar-admin`](reference-pulsar-admin) CLI tool. +Since Pulsar IO connectors are running as [Pulsar Functions](functions-overview.md), so you can use [`functions`](reference-pulsar-admin.md#source) commands +available in the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. ### Retrieve Connector Metadata diff --git a/site2/website/versioned_docs/version-2.3.2/io-overview.md b/site2/website/versioned_docs/version-2.3.2/io-overview.md index 2a6ecaee5883f..ea49e4b042e8c 100644 --- a/site2/website/versioned_docs/version-2.3.2/io-overview.md +++ b/site2/website/versioned_docs/version-2.3.2/io-overview.md @@ -157,7 +157,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.3.2/io-quickstart.md b/site2/website/versioned_docs/version-2.3.2/io-quickstart.md index c4230323e66f4..83a57170dfc91 100644 --- a/site2/website/versioned_docs/version-2.3.2/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.3.2/io-quickstart.md @@ -6,14 +6,14 @@ original_id: io-quickstart --- This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O in tandem with running the steps in this guide +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O in tandem with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you will be able to: - Connect your Pulsar cluster with your Cassandra cluster :::tip -1. These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However all +1. These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However all the commands used in this tutorial should be able to be used in a multi-nodes Pulsar cluster without any changes. 2. All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -260,11 +260,11 @@ configs: ``` -To learn more about Cassandra Connector, see [Cassandra Connector](io-cassandra). +To learn more about Cassandra Connector, see [Cassandra Connector](io-cassandra.md). ### Submit a Cassandra Sink -Pulsar provides the [CLI](reference-cli-tools) for running and managing Pulsar I/O connectors. +Pulsar provides the [CLI](reference-cli-tools.md) for running and managing Pulsar I/O connectors. We can run following command to sink a sink connector with type `cassandra` and config file `examples/cassandra-sink.yml`. diff --git a/site2/website/versioned_docs/version-2.3.2/pulsar-2.0.md b/site2/website/versioned_docs/version-2.3.2/pulsar-2.0.md index 560c8c1bd7cb7..07fba365e9ff2 100644 --- a/site2/website/versioned_docs/version-2.3.2/pulsar-2.0.md +++ b/site2/website/versioned_docs/version-2.3.2/pulsar-2.0.md @@ -4,13 +4,13 @@ title: Pulsar 2.0 sidebar_label: "Pulsar 2.0" --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.3.2/reference-cli-tools.md b/site2/website/versioned_docs/version-2.3.2/reference-cli-tools.md index 025fb6f2a07cb..6c04428c541c1 100644 --- a/site2/website/versioned_docs/version-2.3.2/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.3.2/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.3.2/security-athenz.md b/site2/website/versioned_docs/version-2.3.2/security-athenz.md index ba27ba43eb9d8..6d338fc4013bd 100644 --- a/site2/website/versioned_docs/version-2.3.2/security-athenz.md +++ b/site2/website/versioned_docs/version-2.3.2/security-athenz.md @@ -75,7 +75,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.3.2/security-authorization.md b/site2/website/versioned_docs/version-2.3.2/security-authorization.md index d9d33665d066b..c5e3c04b03524 100644 --- a/site2/website/versioned_docs/version-2.3.2/security-authorization.md +++ b/site2/website/versioned_docs/version-2.3.2/security-authorization.md @@ -26,7 +26,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -58,7 +58,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -86,7 +86,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication @@ -116,7 +116,7 @@ PulsarAdmin admin = PulsarAdmin.builder() When a client is identified with multiple roles in a token (the type of role claim in the token is an array) during the authentication process, Pulsar supports to check the permissions of all the roles and further authorize the client as long as one of its roles has the required permissions. > **Note**
    -> This authorization method is only compatible with [JWT authentication](security-jwt). +> This authorization method is only compatible with [JWT authentication](security-jwt.md). To enable this authorization method, configure the authorization provider as `MultiRolesTokenAuthorizationProvider` in the `conf/broker.conf` file. diff --git a/site2/website/versioned_docs/version-2.3.2/security-kerberos.md b/site2/website/versioned_docs/version-2.3.2/security-kerberos.md index 15c23a15b2e03..2936ad0723927 100644 --- a/site2/website/versioned_docs/version-2.3.2/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.3.2/security-kerberos.md @@ -301,7 +301,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, the authenticated principal is used as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, the authenticated principal is used as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). ## Regarding authorization between BookKeeper and ZooKeeper diff --git a/site2/website/versioned_docs/version-2.3.2/security-overview.md b/site2/website/versioned_docs/version-2.3.2/security-overview.md index 0b40dceba0f8c..ad9a809e9125a 100644 --- a/site2/website/versioned_docs/version-2.3.2/security-overview.md +++ b/site2/website/versioned_docs/version-2.3.2/security-overview.md @@ -22,7 +22,7 @@ In Pulsar, a *role* is a string, like `admin` or `app1`, that can represent a si to produce or consume from certain topics, administer the configuration for tenants, and more. Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign that client a *role token*. This -role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication Providers @@ -34,7 +34,7 @@ Currently Pulsar supports two authentication providers: ## Contents -- [Encryption](security-tls-transport.md) and [Authentication](security-tls-authentication) using TLS +- [Encryption](security-tls-transport.md) and [Authentication](security-tls-authentication.md) using TLS - [Authentication using Athenz](security-athenz.md) - [Authentication using Kerberos](security-kerberos.md) - [Authorization and ACLs](security-authorization.md) diff --git a/site2/website/versioned_docs/version-2.3.2/security-tls-authentication.md b/site2/website/versioned_docs/version-2.3.2/security-tls-authentication.md index e6921bd44181f..d34ecbc1fdcf3 100644 --- a/site2/website/versioned_docs/version-2.3.2/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.3.2/security-tls-authentication.md @@ -6,9 +6,9 @@ sidebar_label: "Authentication using TLS" ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -102,7 +102,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -122,7 +122,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.3.2/security-tls-transport.md b/site2/website/versioned_docs/version-2.3.2/security-tls-transport.md index d4a36de0a3dcd..51e90aa766080 100644 --- a/site2/website/versioned_docs/version-2.3.2/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.3.2/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text, which means that all data is sent in the clear. TLS can be used to encrypt this traffic so that it cannot be snooped by a man-in-the-middle attacker. -TLS can be configured for both encryption and authentication. You may configure just TLS transport encryption, which is covered in this guide. TLS authentication is covered [elsewhere](security-tls-authentication.md). Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +TLS can be configured for both encryption and authentication. You may configure just TLS transport encryption, which is covered in this guide. TLS authentication is covered [elsewhere](security-tls-authentication.md). Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may have a performance impact due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -A third kind of key pair, **client key pairs**, are used for [client authentication](security-tls-authentication). +A third kind of key pair, **client key pairs**, are used for [client authentication](security-tls-authentication.md). The **certificate authority** private key should be stored in a very secure location (a fully encrypted, disconnected, air gapped computer). The certificate authority public key, the **trust cert**, can be freely shared. @@ -27,7 +27,7 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server they are talking to has a key pair that was signed by the certificate authority. A man-in-the-middle attacker would not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that was signed by the certificate authority. The Common Name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that was signed by the certificate authority. The Common Name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). ## Creating TLS Certificates @@ -128,7 +128,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Broker Configuration -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you'll need to make some changes to `broker.conf`, which is located in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you'll need to make some changes to `broker.conf`, which is located in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -199,7 +199,7 @@ The examples below show hostname verification being disabled for the Java client ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You'll need to add the following parameters to that file to use TLS transport with Pulsar's CLI tools: diff --git a/site2/website/versioned_docs/version-2.3.2/security-token-admin.md b/site2/website/versioned_docs/version-2.3.2/security-token-admin.md index 4accb85a75ae0..a4370fb3f8190 100644 --- a/site2/website/versioned_docs/version-2.3.2/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.3.2/security-token-admin.md @@ -162,7 +162,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.3.2/security-token-client.md b/site2/website/versioned_docs/version-2.3.2/security-token-client.md index b955bcddacf7d..2ef64e045e3b7 100644 --- a/site2/website/versioned_docs/version-2.3.2/security-token-client.md +++ b/site2/website/versioned_docs/version-2.3.2/security-token-client.md @@ -24,12 +24,12 @@ eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJKb2UifQ.ipevRNuRP6HflG8cFKnmUPtypruRC4fb1DWtoLL6 Application specifies the token when you are creating the client instance. An alternative is to pass a "token supplier" (a function that returns the token when the client library needs one). -See [Token authentication admin](security-token-admin) for a reference on how to enable token +See [Token authentication admin](security-token-admin.md) for a reference on how to enable token authentication on a Pulsar cluster. ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.3.2/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.3.2/sql-deployment-configurations.md index 639ac988b3409..069c6ac9c92da 100644 --- a/site2/website/versioned_docs/version-2.3.2/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.3.2/sql-deployment-configurations.md @@ -141,7 +141,7 @@ Since Pulsar SQL is powered by [Trino (formerly Presto SQL)](https://trino.io), :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.3.2/sql-overview.md b/site2/website/versioned_docs/version-2.3.2/sql-overview.md index 2f827f4267933..5b01dfde56e7f 100644 --- a/site2/website/versioned_docs/version-2.3.2/sql-overview.md +++ b/site2/website/versioned_docs/version-2.3.2/sql-overview.md @@ -4,7 +4,7 @@ title: Pulsar SQL Overview sidebar_label: "Overview" --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL)](https://trino.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL.md)](https://trino.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.3.2/standalone-docker.md b/site2/website/versioned_docs/version-2.3.2/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.3.2/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.3.2/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.3.2/standalone.md b/site2/website/versioned_docs/version-2.3.2/standalone.md index c05b94a5aebdc..486548b5bdc0d 100644 --- a/site2/website/versioned_docs/version-2.3.2/standalone.md +++ b/site2/website/versioned_docs/version-2.3.2/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -49,9 +49,9 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](reference-pulsar-admin). +`bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](reference-pulsar-admin.md). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -60,7 +60,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. #### Install other optional components @@ -111,7 +111,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -125,7 +125,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -158,12 +158,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: diff --git a/site2/website/versioned_docs/version-2.4.0/admin-api-clusters.md b/site2/website/versioned_docs/version-2.4.0/admin-api-clusters.md index 04db2f60759bc..1255ad25d7aac 100644 --- a/site2/website/versioned_docs/version-2.4.0/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.4.0/admin-api-clusters.md @@ -88,7 +88,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -112,11 +112,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block > - For the latest and complete information about `Java admin API`, including classes, methods, descriptions, and more, see [Java admin API doc](https://pulsar.apache.org/api/admin/). -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -121,7 +121,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block For a guide to managing message persistence, retention, and expiry in Pulsar, see [this cookbook](cookbooks-retention-expiry). +> For a guide to managing message persistence, retention, and expiry in Pulsar, see [this cookbook](cookbooks-retention-expiry.md). ### Deploying BookKeeper diff --git a/site2/website/versioned_docs/version-2.4.0/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.4.0/client-libraries-cpp.md index 5ba9e03d89521..b709463ff0309 100644 --- a/site2/website/versioned_docs/version-2.4.0/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.4.0/client-libraries-cpp.md @@ -661,7 +661,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema This section describes some examples about schema. For more information about -schema, see [Pulsar schema](schema-get-started). +schema, see [Pulsar schema](schema-get-started.md). ### Avro schema diff --git a/site2/website/versioned_docs/version-2.4.0/client-libraries-go.md b/site2/website/versioned_docs/version-2.4.0/client-libraries-go.md index 5ff15ac7007f8..486d7bb8404c2 100644 --- a/site2/website/versioned_docs/version-2.4.0/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.4.0/client-libraries-go.md @@ -16,7 +16,7 @@ The Pulsar Go client can be used to create Pulsar [producers](#producers), [cons ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Installing go package @@ -50,7 +50,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -68,7 +68,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -238,14 +238,14 @@ Parameter | Description | Default :---------|:------------|:------- `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | | `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | ## Consumers @@ -555,7 +555,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.4.0/client-libraries-java.md b/site2/website/versioned_docs/version-2.4.0/client-libraries-java.md index 778129d90777b..53119f6bef44c 100644 --- a/site2/website/versioned_docs/version-2.4.0/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.4.0/client-libraries-java.md @@ -5,16 +5,16 @@ sidebar_label: "Java" original_id: client-libraries-java --- -The Pulsar Java client can be used both to create Java producers, consumers, and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +The Pulsar Java client can be used both to create Java producers, consumers, and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. Javadoc for the Pulsar client is divided up into two domains, by package: Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) -This document will focus only on the client API for producing and consuming messages on Pulsar topics. For a guide to using the Java admin client, see [The Pulsar admin interface](admin-api-overview). +This document will focus only on the client API for producing and consuming messages on Pulsar topics. For a guide to using the Java admin client, see [The Pulsar admin interface](admin-api-overview.md). ## Installation @@ -54,7 +54,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -80,7 +80,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -111,7 +111,7 @@ PulsarClient client = PulsarClient.builder() ``` > #### Default broker URLs for standalone clusters -> If you're running a cluster in [standalone mode](getting-started-standalone), the broker will be available at the `pulsar://localhost:6650` URL by default. +> If you're running a cluster in [standalone mode](getting-started-standalone.md), the broker will be available at the `pulsar://localhost:6650` URL by default. Check out the Javadoc for the {@inject: javadoc:PulsarClient:/client/org/apache/pulsar/client/api/PulsarClient} class for a full listing of configurable parameters. @@ -188,7 +188,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -604,7 +604,7 @@ The code sample above shows pointing the `Reader` object to a specific message ( ## Schemas -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](concepts-schema-registry) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. Here's an example: +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](concepts-schema-registry.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. Here's an example: ```java @@ -699,7 +699,7 @@ The following schema formats are currently available for Java: ## Authentication -Pulsar currently supports two authentication schemes: [TLS](security-tls-authentication.md) and [Athenz](security-athenz). The Pulsar Java client can be used with both. +Pulsar currently supports two authentication schemes: [TLS](security-tls-authentication.md) and [Athenz](security-athenz.md). The Pulsar Java client can be used with both. ### TLS Authentication @@ -726,7 +726,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` diff --git a/site2/website/versioned_docs/version-2.4.0/client-libraries-python.md b/site2/website/versioned_docs/version-2.4.0/client-libraries-python.md index 17dc69290260c..f61cedffe967b 100644 --- a/site2/website/versioned_docs/version-2.4.0/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.4.0/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -The Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +The Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. ## Installation diff --git a/site2/website/versioned_docs/version-2.4.0/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.4.0/client-libraries-websocket.md index a6e60361749de..0b31ffe78161a 100644 --- a/site2/website/versioned_docs/version-2.4.0/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.4.0/client-libraries-websocket.md @@ -4,14 +4,14 @@ title: Pulsar WebSocket API sidebar_label: "WebSocket" --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.4.0/client-libraries.md b/site2/website/versioned_docs/version-2.4.0/client-libraries.md index 536cd0ccda82f..8f91dfbbbdd8c 100644 --- a/site2/website/versioned_docs/version-2.4.0/client-libraries.md +++ b/site2/website/versioned_docs/version-2.4.0/client-libraries.md @@ -8,12 +8,12 @@ Pulsar supports the following client libraries: |Language|Documentation|Release note|Code repo |---|---|---|--- -Java |- [User doc](client-libraries-java)

    - [API doc](https://pulsar.apache.org/api/client/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client) -C++ | - [User doc](client-libraries-cpp)

    - [API doc](https://pulsar.apache.org/api/cpp/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp) -Python | - [User doc](client-libraries-python)

    - [API doc](https://pulsar.apache.org/api/python/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) -WebSocket| [User doc](client-libraries-websocket) | [Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-websocket) +Java |- [User doc](client-libraries-java.md)

    - [API doc](https://pulsar.apache.org/api/client/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client) +C++ | - [User doc](client-libraries-cpp.md)

    - [API doc](https://pulsar.apache.org/api/cpp/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp) +Python | - [User doc](client-libraries-python.md)

    - [API doc](https://pulsar.apache.org/api/python/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) +WebSocket| [User doc](client-libraries-websocket.md) | [Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-websocket) Go client|[User doc](client-libraries-go.md)|[Here](https://github.com/apache/pulsar-client-go/blob/master/CHANGELOG) |[Here](https://github.com/apache/pulsar-client-go) -Node.js|[User doc](client-libraries-node)|[Here](https://github.com/apache/pulsar-client-node/releases) |[Here](https://github.com/apache/pulsar-client-node) +Node.js|[User doc](client-libraries-node.md)|[Here](https://github.com/apache/pulsar-client-node/releases) |[Here](https://github.com/apache/pulsar-client-node) C# |[User doc](client-libraries-dotnet.md)| [Here](https://github.com/apache/pulsar-dotpulsar/blob/master/CHANGELOG)|[Here](https://github.com/apache/pulsar-dotpulsar) :::note diff --git a/site2/website/versioned_docs/version-2.4.0/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.4.0/concepts-architecture-overview.md index 0d96b59f8ae91..1ce346fc16941 100644 --- a/site2/website/versioned_docs/version-2.4.0/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.4.0/concepts-architecture-overview.md @@ -4,7 +4,7 @@ title: Architecture Overview sidebar_label: "Architecture" --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -16,20 +16,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers. The producers connect to the brokers to publish messages and the consumers connect to the brokers to consume the messages. -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -39,9 +39,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -139,17 +139,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -157,7 +157,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.4.0/concepts-authentication.md b/site2/website/versioned_docs/version-2.4.0/concepts-authentication.md index 335da8dc5a519..fbefead69f382 100644 --- a/site2/website/versioned_docs/version-2.4.0/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.4.0/concepts-authentication.md @@ -4,5 +4,5 @@ title: Authentication and Authorization sidebar_label: "Authentication and Authorization" --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.4.0/concepts-clients.md b/site2/website/versioned_docs/version-2.4.0/concepts-clients.md index 65201b5ce1cd2..c0fe61cd06e76 100644 --- a/site2/website/versioned_docs/version-2.4.0/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.4.0/concepts-clients.md @@ -4,12 +4,12 @@ title: Pulsar Clients sidebar_label: "Clients" --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -37,7 +37,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.4.0/concepts-messaging.md b/site2/website/versioned_docs/version-2.4.0/concepts-messaging.md index a4496d7727fcc..900a1dd89f5f4 100644 --- a/site2/website/versioned_docs/version-2.4.0/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.4.0/concepts-messaging.md @@ -24,7 +24,7 @@ Publish time | The timestamp of when the message was published (automatically ap Event time | An optional timestamp that applications can attach to the message representing when something happened, e.g. when the message was processed. The event time of a message is 0 if none is explicitly set. -> For a more in-depth breakdown of Pulsar message contents, see the documentation on Pulsar's [binary protocol](developing-binary-protocol). +> For a more in-depth breakdown of Pulsar message contents, see the documentation on Pulsar's [binary protocol](developing-binary-protocol.md). ## Producers @@ -67,7 +67,7 @@ Messages can be received from [brokers](reference-terminology.md#broker) either ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -269,7 +269,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -281,7 +281,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -319,7 +319,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -374,7 +374,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -397,12 +397,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, which means that you don't need to modify your Pulsar client code. Instead, you only need to make administrative changes (see the [Managing message deduplication](cookbooks-deduplication) cookbook for a guide). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, which means that you don't need to modify your Pulsar client code. Instead, you only need to make administrative changes (see the [Managing message deduplication](cookbooks-deduplication.md) cookbook for a guide). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.4.0/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.4.0/concepts-multi-tenancy.md index 8a17e72a4c8b3..26bd6a2fd183f 100644 --- a/site2/website/versioned_docs/version-2.4.0/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.4.0/concepts-multi-tenancy.md @@ -4,7 +4,7 @@ title: Multi Tenancy sidebar_label: "Multi Tenancy" --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -20,7 +20,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -28,7 +28,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.4.0/concepts-overview.md b/site2/website/versioned_docs/version-2.4.0/concepts-overview.md index 119d46508ed28..bf4993a3110b5 100644 --- a/site2/website/versioned_docs/version-2.4.0/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.4.0/concepts-overview.md @@ -8,15 +8,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.4.0/concepts-replication.md b/site2/website/versioned_docs/version-2.4.0/concepts-replication.md index d4653ca7aecd8..9cc3a7c3415bd 100644 --- a/site2/website/versioned_docs/version-2.4.0/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.4.0/concepts-replication.md @@ -8,7 +8,7 @@ Regardless of industries, when an unforeseen event occurs and brings day-to-day Pulsar's geo-replication mechanism is typically used for disaster recovery, enabling the replication of persistently stored message data across multiple data centers. For instance, your application is publishing data in one region and you would like to process it for consumption in other regions. With Pulsar’s geo-replication mechanism, messages can be produced and consumed in different geo-locations. -The diagram below illustrates the process of [geo-replication](administration-geo). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. +The diagram below illustrates the process of [geo-replication](administration-geo.md). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. ![A typical geo-replication example with full-mesh pattern](/assets/full-mesh-replication.svg) diff --git a/site2/website/versioned_docs/version-2.4.0/concepts-schema-registry.md b/site2/website/versioned_docs/version-2.4.0/concepts-schema-registry.md index 72cb46662cbb2..c39f384601410 100644 --- a/site2/website/versioned_docs/version-2.4.0/concepts-schema-registry.md +++ b/site2/website/versioned_docs/version-2.4.0/concepts-schema-registry.md @@ -17,7 +17,7 @@ Both approaches are available in Pulsar, and you're free to adopt one or the oth #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-go.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-go.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ## Basic architecture @@ -47,7 +47,7 @@ Schemas are versioned in succession.The schema is stored in a broker that handle Once a version is assigned/fetched to/for a schema, all subsequent messages produced by that producer are tagged with the appropriate version. -In order to illustrate how schema versioning works, let's walk through an example. Imagine that the Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begin sending messages: +In order to illustrate how schema versioning works, let's walk through an example. Imagine that the Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begin sending messages: ```java diff --git a/site2/website/versioned_docs/version-2.4.0/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.4.0/concepts-tiered-storage.md index 27cb6c0d32bd1..9e2eac528b70f 100644 --- a/site2/website/versioned_docs/version-2.4.0/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.4.0/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3 and Google Cloud Storage (GCS) for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on bookkeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from bookkeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.4.0/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.4.0/concepts-topic-compaction.md index 790ce76b7c5e0..300cf68956dd1 100644 --- a/site2/website/versioned_docs/version-2.4.0/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.4.0/concepts-topic-compaction.md @@ -6,7 +6,7 @@ sidebar_label: "Topic Compaction" Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -23,7 +23,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.4.0/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.4.0/cookbooks-compaction.md index d1a3bafffb580..58ee52a622eb7 100644 --- a/site2/website/versioned_docs/version-2.4.0/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.4.0/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually {#trigger} -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.4.0/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.4.0/cookbooks-non-persistent.md index d40c4fbd0cf01..1346d19a04263 100644 --- a/site2/website/versioned_docs/version-2.4.0/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.4.0/cookbooks-non-persistent.md @@ -40,7 +40,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -54,7 +54,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.4.0/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.4.0/cookbooks-partitioned.md index 25896934788fc..c857d019edc02 100644 --- a/site2/website/versioned_docs/version-2.4.0/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.4.0/cookbooks-partitioned.md @@ -3,4 +3,4 @@ id: cookbooks-partitioned title: Partitioned topics sidebar_label: "Partitioned Topics" --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.4.0/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.4.0/cookbooks-retention-expiry.md index 86160c3f7ecf8..cd764fad6eb37 100644 --- a/site2/website/versioned_docs/version-2.4.0/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.4.0/cookbooks-retention-expiry.md @@ -21,7 +21,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.4.0/deploy-aws.md b/site2/website/versioned_docs/version-2.4.0/deploy-aws.md index 5db846fe790f2..60ac4c22d7672 100644 --- a/site2/website/versioned_docs/version-2.4.0/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.4.0/deploy-aws.md @@ -4,7 +4,7 @@ title: Deploying a Pulsar cluster on AWS using Terraform and Ansible sidebar_label: "Amazon Web Services" --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -209,7 +209,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.4.0/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.4.0/deploy-bare-metal-multi-cluster.md index 70dff45080b03..8a765d8843f43 100644 --- a/site2/website/versioned_docs/version-2.4.0/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.4.0/deploy-bare-metal-multi-cluster.md @@ -6,13 +6,13 @@ sidebar_label: "Bare metal multi-cluster" :::tip -1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster). -2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster.md). +2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo).Deploying a multi-cluster Pulsar instance consists of the following steps: +A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md).Deploying a multi-cluster Pulsar instance consists of the following steps: 1. Deploying two separate ZooKeeper quorums: a local quorum for each cluster in the instance and a configuration store quorum for instance-wide tasks 2. Initializing cluster metadata for each cluster @@ -21,7 +21,7 @@ A Pulsar instance consists of multiple Pulsar clusters working in unison. You ca > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. ## System requirement @@ -65,7 +65,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -244,7 +244,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -363,16 +363,16 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. -You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > **Service discovery already provided by many scheduling systems** -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.4.0/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.4.0/deploy-bare-metal.md index 0da5f4d027dae..0f9e17adacef2 100644 --- a/site2/website/versioned_docs/version-2.4.0/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.4.0/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you're interested in experimenting with Pulsar or using it in a startup or on a single team, we recommend opting for a single cluster. If you do need to run a multi-cluster Pulsar instance, -however, see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +however, see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and make sure it is installed under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and make sure it is installed under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you could reference this [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you could reference this [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -266,9 +266,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (we don't recommend using a different port). -`--web-service-url-tls` | If you're using [TLS](security-tls-transport), you'll also need to specify a TLS web service URL for the cluster. The default port is 8443 (we don't recommend using a different port). +`--web-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster. The default port is 8443 (we don't recommend using a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (we don't recommend using a different port). -`--broker-service-url-tls` | If you're using [TLS](security-tls-transport), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (we don't recommend using a different port). +`--broker-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (we don't recommend using a different port). ## Deploying a BookKeeper cluster @@ -378,7 +378,7 @@ webServicePortTls=8443 ### Enabling Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -396,7 +396,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you would like to learn more options about deploying functions worker, please checkout [Deploy and manage functions worker](functions-worker). +If you would like to learn more options about deploying functions worker, please checkout [Deploy and manage functions worker](functions-worker.md). ### Starting Brokers diff --git a/site2/website/versioned_docs/version-2.4.0/deploy-dcos.md b/site2/website/versioned_docs/version-2.4.0/deploy-dcos.md index 07f446edaffd6..599d3a1b0454c 100644 --- a/site2/website/versioned_docs/version-2.4.0/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.4.0/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -To enable all built-in [Pulsar IO](io-overview) connectors in your Pulsar deploymente, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). +To enable all built-in [Pulsar IO](io-overview.md) connectors in your Pulsar deploymente, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.4.0/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.4.0/deploy-kubernetes.md index 4e170dcd4711d..7455247c901d4 100644 --- a/site2/website/versioned_docs/version-2.4.0/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.4.0/deploy-kubernetes.md @@ -5,6 +5,6 @@ sidebar_label: "Kubernetes" --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.4.0/deploy-monitoring.md b/site2/website/versioned_docs/version-2.4.0/deploy-monitoring.md index adf3587b7aeb3..d581fe2edfacb 100644 --- a/site2/website/versioned_docs/version-2.4.0/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.4.0/deploy-monitoring.md @@ -112,7 +112,7 @@ http://$FUNCTIONS_WORKER_ADDRESS:$WORKER_PORT/metrics: You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy). +When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy.md). ## Dashboards @@ -120,7 +120,7 @@ When you collect time series statistics, the major problem is to make sure the n ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager). +The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.4.0/develop-schema.md b/site2/website/versioned_docs/version-2.4.0/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.4.0/develop-schema.md +++ b/site2/website/versioned_docs/version-2.4.0/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.4.0/functions-api.md b/site2/website/versioned_docs/version-2.4.0/functions-api.md index 21608eaed32b4..6409c0556285c 100644 --- a/site2/website/versioned_docs/version-2.4.0/functions-api.md +++ b/site2/website/versioned_docs/version-2.4.0/functions-api.md @@ -5,9 +5,9 @@ sidebar_label: "API" original_id: functions-api --- -[Pulsar Functions](functions-overview) provides an easy-to-use API that developers can use to create and manage processing logic for the Apache Pulsar messaging system. With Pulsar Functions, you can write functions of any level of complexity in [Java](#functions-for-java) or [Python](#functions-for-python) and run them in conjunction with a Pulsar cluster without needing to run a separate stream processing engine. +[Pulsar Functions](functions-overview.md) provides an easy-to-use API that developers can use to create and manage processing logic for the Apache Pulsar messaging system. With Pulsar Functions, you can write functions of any level of complexity in [Java](#functions-for-java) or [Python](#functions-for-python) and run them in conjunction with a Pulsar cluster without needing to run a separate stream processing engine. -> For a more in-depth overview of the Pulsar Functions feature, see the [Pulsar Functions overview](functions-overview). +> For a more in-depth overview of the Pulsar Functions feature, see the [Pulsar Functions overview](functions-overview.md). ## Core programming model @@ -45,7 +45,7 @@ Some things to note about this Pulsar Function: ### Example deployment -Deploying Pulsar Functions is handled by the [`pulsar-admin`](reference-pulsar-admin) CLI tool, in particular the [`functions`](reference-pulsar-admin.md#functions) command. Here's an example command that would run our [sanitizer](#example-function) function from above in [local run](functions-deploying.md#local-run-mode) mode: +Deploying Pulsar Functions is handled by the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool, in particular the [`functions`](reference-pulsar-admin.md#functions) command. Here's an example command that would run our [sanitizer](#example-function) function from above in [local run](functions-deploying.md#local-run-mode) mode: ```bash @@ -61,7 +61,7 @@ $ bin/pulsar-admin functions localrun \ ``` -For instructions on running functions in your Pulsar cluster, see the [Deploying Pulsar Functions](functions-deploying) guide. +For instructions on running functions in your Pulsar cluster, see the [Deploying Pulsar Functions](functions-deploying.md) guide. ### Available APIs @@ -101,7 +101,7 @@ Pulsar Functions can take byte arrays as inputs and spit out byte arrays as outp * [SerDe](#SerDe) ### Schema Registry -Pulsar has a built in [Schema Registry](concepts-schema-registry) and comes bundled with a variety of popular schema types(avro, json and protobuf). Pulsar Functions can leverage existing schema information from input topics to derive the input type. The same applies for output topic as well. +Pulsar has a built in [Schema Registry](concepts-schema-registry.md) and comes bundled with a variety of popular schema types(avro, json and protobuf.md). Pulsar Functions can leverage existing schema information from input topics to derive the input type. The same applies for output topic as well. ### SerDe @@ -560,7 +560,7 @@ public class MetricRecorderFunction implements Function { ``` -> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring) guide. +> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring.md) guide. ## Functions for Python @@ -572,7 +572,7 @@ Writing Pulsar Functions in Python entails implementing one of two things: ### Get started -Regardless of which [deployment mode](functions-deploying) you're using, 'pulsar-client' python library has to installed on any machine that's running Pulsar Functions written in Python. +Regardless of which [deployment mode](functions-deploying.md) you're using, 'pulsar-client' python library has to installed on any machine that's running Pulsar Functions written in Python. That could be your local machine for [local run mode](functions-deploying.md#local-run-mode) or a machine running a Pulsar [broker](reference-terminology.md#broker) for [cluster mode](functions-deploying.md#cluster-mode). To install those libraries using pip: diff --git a/site2/website/versioned_docs/version-2.4.0/functions-overview.md b/site2/website/versioned_docs/version-2.4.0/functions-overview.md index ceb87106001e0..d860e1bc37cd5 100644 --- a/site2/website/versioned_docs/version-2.4.0/functions-overview.md +++ b/site2/website/versioned_docs/version-2.4.0/functions-overview.md @@ -176,7 +176,7 @@ class RoutingFunction(Function): ## Command-line interface -Pulsar Functions are managed using the [`pulsar-admin`](reference-pulsar-admin) CLI tool (in particular the [`functions`](reference-pulsar-admin.md#functions) command). The following example runs a function in the [local run mode](#local-run-mode). +Pulsar Functions are managed using the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool (in particular the [`functions`](reference-pulsar-admin.md#functions) command). The following example runs a function in the [local run mode](#local-run-mode). ```bash @@ -567,7 +567,7 @@ $ bin/pulsar-admin functions create \ ``` ## Metrics -Pulsar Functions that use [Pulsar Functions SDK](#the-pulsar-functions-sdk) can publish metrics to Pulsar. For more information, see [Metrics for Pulsar Functions](functions-metrics). +Pulsar Functions that use [Pulsar Functions SDK](#the-pulsar-functions-sdk) can publish metrics to Pulsar. For more information, see [Metrics for Pulsar Functions](functions-metrics.md). ## State storage Pulsar Functions use [Apache BookKeeper](https://bookkeeper.apache.org) as a state storage interface. Pulsar installation, including the local standalone installation, includes deployment of BookKeeper bookies. \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.4.0/functions-worker.md b/site2/website/versioned_docs/version-2.4.0/functions-worker.md index e7740ad9ad10f..15faf18c99bca 100644 --- a/site2/website/versioned_docs/version-2.4.0/functions-worker.md +++ b/site2/website/versioned_docs/version-2.4.0/functions-worker.md @@ -123,7 +123,7 @@ tlsTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). **Enable Authentication Provider** @@ -150,7 +150,7 @@ properties: ``` For *Token Authentication* prodivder, add necessary settings under `properties` if needed. -See [Token Authentication](security-token-admin) for more details. +See [Token Authentication](security-token-admin.md) for more details. ``` diff --git a/site2/website/versioned_docs/version-2.4.0/getting-started-docker.md b/site2/website/versioned_docs/version-2.4.0/getting-started-docker.md index 26908245cb2c9..904d67401668a 100644 --- a/site2/website/versioned_docs/version-2.4.0/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.4.0/getting-started-docker.md @@ -43,8 +43,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -103,7 +103,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.4.0/getting-started-standalone.md b/site2/website/versioned_docs/version-2.4.0/getting-started-standalone.md index 28b04bcc19203..fd7edad4a316b 100644 --- a/site2/website/versioned_docs/version-2.4.0/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.4.0/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -51,9 +51,9 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](reference-pulsar-admin). +`bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](reference-pulsar-admin.md). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -62,7 +62,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -111,7 +111,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -125,7 +125,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -158,12 +158,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: diff --git a/site2/website/versioned_docs/version-2.4.0/io-cdc.md b/site2/website/versioned_docs/version-2.4.0/io-cdc.md index df373977e3eaa..a3527dc0a9d65 100644 --- a/site2/website/versioned_docs/version-2.4.0/io-cdc.md +++ b/site2/website/versioned_docs/version-2.4.0/io-cdc.md @@ -12,8 +12,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.4.0/io-develop.md b/site2/website/versioned_docs/version-2.4.0/io-develop.md index 69c9a2f42c428..d361a302b2529 100644 --- a/site2/website/versioned_docs/version-2.4.0/io-develop.md +++ b/site2/website/versioned_docs/version-2.4.0/io-develop.md @@ -13,15 +13,15 @@ import TabItem from '@theme/TabItem'; This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -274,7 +274,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -305,7 +305,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). @@ -380,7 +380,7 @@ Pulsar connectors enable you to move data in and out of Pulsar easily. It is imp - Check the metrics provided by Pulsar. - Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. - Set and check your customized metrics. diff --git a/site2/website/versioned_docs/version-2.4.0/io-managing.md b/site2/website/versioned_docs/version-2.4.0/io-managing.md index cce795bbae86a..8c834410a5d0f 100644 --- a/site2/website/versioned_docs/version-2.4.0/io-managing.md +++ b/site2/website/versioned_docs/version-2.4.0/io-managing.md @@ -50,7 +50,7 @@ For details, consult the documentation for [individual connectors](io-overview.m ## Running Connectors -Pulsar connectors can be managed using the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands of the [`pulsar-admin`](reference-pulsar-admin) CLI tool. +Pulsar connectors can be managed using the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands of the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. ### Running sources @@ -166,8 +166,8 @@ Here's an example to submit a Cassandra sink: ## Monitoring Connectors -Since Pulsar IO connectors are running as [Pulsar Functions](functions-overview), so you can use [`functions`](reference-pulsar-admin.md#source) commands -available in the [`pulsar-admin`](reference-pulsar-admin) CLI tool. +Since Pulsar IO connectors are running as [Pulsar Functions](functions-overview.md), so you can use [`functions`](reference-pulsar-admin.md#source) commands +available in the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. ### Retrieve Connector Metadata diff --git a/site2/website/versioned_docs/version-2.4.0/io-overview.md b/site2/website/versioned_docs/version-2.4.0/io-overview.md index 2a6ecaee5883f..ea49e4b042e8c 100644 --- a/site2/website/versioned_docs/version-2.4.0/io-overview.md +++ b/site2/website/versioned_docs/version-2.4.0/io-overview.md @@ -157,7 +157,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.4.0/io-quickstart.md b/site2/website/versioned_docs/version-2.4.0/io-quickstart.md index acab1ebfe4321..5bb9750f69dd6 100644 --- a/site2/website/versioned_docs/version-2.4.0/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.4.0/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you will be able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you will be able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However all the commands used in this tutorial should be able to be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -214,11 +214,11 @@ configs: ``` -To learn more about Cassandra Connector, see [Cassandra Connector](io-cassandra). +To learn more about Cassandra Connector, see [Cassandra Connector](io-cassandra.md). ### Submit a Cassandra sink -Pulsar provides the [CLI](reference-cli-tools) for running and managing Pulsar I/O connectors. +Pulsar provides the [CLI](reference-cli-tools.md) for running and managing Pulsar I/O connectors. We can run following command to sink a sink connector with type `cassandra` and config file `examples/cassandra-sink.yml`. @@ -582,7 +582,7 @@ Now that we have a MySQL running locally. In this section, we will configure a J ### Submit a JDBC sink -Pulsar provides the [CLI](admin-api-overview) for running and managing Pulsar I/O connectors. +Pulsar provides the [CLI](admin-api-overview.md) for running and managing Pulsar I/O connectors. This example creates a sink connector and specifies the desired information. diff --git a/site2/website/versioned_docs/version-2.4.0/pulsar-2.0.md b/site2/website/versioned_docs/version-2.4.0/pulsar-2.0.md index 560c8c1bd7cb7..07fba365e9ff2 100644 --- a/site2/website/versioned_docs/version-2.4.0/pulsar-2.0.md +++ b/site2/website/versioned_docs/version-2.4.0/pulsar-2.0.md @@ -4,13 +4,13 @@ title: Pulsar 2.0 sidebar_label: "Pulsar 2.0" --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.4.0/reference-cli-tools.md b/site2/website/versioned_docs/version-2.4.0/reference-cli-tools.md index aa12b21430199..2a515374ce71a 100644 --- a/site2/website/versioned_docs/version-2.4.0/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.4.0/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.4.0/security-athenz.md b/site2/website/versioned_docs/version-2.4.0/security-athenz.md index ba27ba43eb9d8..6d338fc4013bd 100644 --- a/site2/website/versioned_docs/version-2.4.0/security-athenz.md +++ b/site2/website/versioned_docs/version-2.4.0/security-athenz.md @@ -75,7 +75,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.4.0/security-authorization.md b/site2/website/versioned_docs/version-2.4.0/security-authorization.md index d9d33665d066b..c5e3c04b03524 100644 --- a/site2/website/versioned_docs/version-2.4.0/security-authorization.md +++ b/site2/website/versioned_docs/version-2.4.0/security-authorization.md @@ -26,7 +26,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -58,7 +58,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -86,7 +86,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication @@ -116,7 +116,7 @@ PulsarAdmin admin = PulsarAdmin.builder() When a client is identified with multiple roles in a token (the type of role claim in the token is an array) during the authentication process, Pulsar supports to check the permissions of all the roles and further authorize the client as long as one of its roles has the required permissions. > **Note**
    -> This authorization method is only compatible with [JWT authentication](security-jwt). +> This authorization method is only compatible with [JWT authentication](security-jwt.md). To enable this authorization method, configure the authorization provider as `MultiRolesTokenAuthorizationProvider` in the `conf/broker.conf` file. diff --git a/site2/website/versioned_docs/version-2.4.0/security-jwt.md b/site2/website/versioned_docs/version-2.4.0/security-jwt.md index 23d6dc95040e4..19eb9d886a4b6 100644 --- a/site2/website/versioned_docs/version-2.4.0/security-jwt.md +++ b/site2/website/versioned_docs/version-2.4.0/security-jwt.md @@ -32,11 +32,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -280,7 +280,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.4.0/security-kerberos.md b/site2/website/versioned_docs/version-2.4.0/security-kerberos.md index 1b2d61ba37ed8..d26a4be927f05 100644 --- a/site2/website/versioned_docs/version-2.4.0/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.4.0/security-kerberos.md @@ -377,7 +377,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, the authenticated principal is used as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, the authenticated principal is used as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enabled authorizationEnabled you need set `superUserRoles` in `broker.conf` that corresponding to the name registered in kdc diff --git a/site2/website/versioned_docs/version-2.4.0/security-overview.md b/site2/website/versioned_docs/version-2.4.0/security-overview.md index f9a623334eb27..ee7d8f9b60b5d 100644 --- a/site2/website/versioned_docs/version-2.4.0/security-overview.md +++ b/site2/website/versioned_docs/version-2.4.0/security-overview.md @@ -17,7 +17,7 @@ You'd better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent one or more clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses the [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses the [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication Providers @@ -30,7 +30,7 @@ Currently Pulsar supports the following authentication providers: ## Contents -- [Encryption](security-tls-transport.md) and [Authentication](security-tls-authentication) using TLS +- [Encryption](security-tls-transport.md) and [Authentication](security-tls-authentication.md) using TLS - [Authentication using Athenz](security-athenz.md) - [Authentication using Kerberos](security-kerberos.md) - [Authorization and ACLs](security-authorization.md) diff --git a/site2/website/versioned_docs/version-2.4.0/security-tls-authentication.md b/site2/website/versioned_docs/version-2.4.0/security-tls-authentication.md index 53159f984a814..140a7bb608172 100644 --- a/site2/website/versioned_docs/version-2.4.0/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.4.0/security-tls-authentication.md @@ -7,7 +7,7 @@ original_id: security-tls-authentication ## TLS Authentication Overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport), but instead of only servers having keys and certs which the client uses to verify the server's identity, clients also have keys and certs which the server uses to verify the client's identity. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md), but instead of only servers having keys and certs which the client uses to verify the server's identity, clients also have keys and certs which the server uses to verify the client's identity. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. ### Creating client certificates @@ -101,7 +101,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, put the following in `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. The role token for this key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. The role token for this key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -121,7 +121,7 @@ When TLS authentication, the client needs to connect via TLS transport, so you n ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You'll need to add the following parameters to that file to use TLS authentication with Pulsar's CLI tools: diff --git a/site2/website/versioned_docs/version-2.4.0/security-tls-transport.md b/site2/website/versioned_docs/version-2.4.0/security-tls-transport.md index d4a36de0a3dcd..51e90aa766080 100644 --- a/site2/website/versioned_docs/version-2.4.0/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.4.0/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text, which means that all data is sent in the clear. TLS can be used to encrypt this traffic so that it cannot be snooped by a man-in-the-middle attacker. -TLS can be configured for both encryption and authentication. You may configure just TLS transport encryption, which is covered in this guide. TLS authentication is covered [elsewhere](security-tls-authentication.md). Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +TLS can be configured for both encryption and authentication. You may configure just TLS transport encryption, which is covered in this guide. TLS authentication is covered [elsewhere](security-tls-authentication.md). Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may have a performance impact due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -A third kind of key pair, **client key pairs**, are used for [client authentication](security-tls-authentication). +A third kind of key pair, **client key pairs**, are used for [client authentication](security-tls-authentication.md). The **certificate authority** private key should be stored in a very secure location (a fully encrypted, disconnected, air gapped computer). The certificate authority public key, the **trust cert**, can be freely shared. @@ -27,7 +27,7 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server they are talking to has a key pair that was signed by the certificate authority. A man-in-the-middle attacker would not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that was signed by the certificate authority. The Common Name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that was signed by the certificate authority. The Common Name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). ## Creating TLS Certificates @@ -128,7 +128,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Broker Configuration -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you'll need to make some changes to `broker.conf`, which is located in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you'll need to make some changes to `broker.conf`, which is located in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -199,7 +199,7 @@ The examples below show hostname verification being disabled for the Java client ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You'll need to add the following parameters to that file to use TLS transport with Pulsar's CLI tools: diff --git a/site2/website/versioned_docs/version-2.4.0/security-token-admin.md b/site2/website/versioned_docs/version-2.4.0/security-token-admin.md index 89e06c2dcd94d..dac79669f0cb7 100644 --- a/site2/website/versioned_docs/version-2.4.0/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.4.0/security-token-admin.md @@ -172,7 +172,7 @@ brokerClientAuthenticationParameters=token:eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ0ZXN0 To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.4.0/security-token-client.md b/site2/website/versioned_docs/version-2.4.0/security-token-client.md index b955bcddacf7d..2ef64e045e3b7 100644 --- a/site2/website/versioned_docs/version-2.4.0/security-token-client.md +++ b/site2/website/versioned_docs/version-2.4.0/security-token-client.md @@ -24,12 +24,12 @@ eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJKb2UifQ.ipevRNuRP6HflG8cFKnmUPtypruRC4fb1DWtoLL6 Application specifies the token when you are creating the client instance. An alternative is to pass a "token supplier" (a function that returns the token when the client library needs one). -See [Token authentication admin](security-token-admin) for a reference on how to enable token +See [Token authentication admin](security-token-admin.md) for a reference on how to enable token authentication on a Pulsar cluster. ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.4.0/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.4.0/sql-deployment-configurations.md index 639ac988b3409..069c6ac9c92da 100644 --- a/site2/website/versioned_docs/version-2.4.0/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.4.0/sql-deployment-configurations.md @@ -141,7 +141,7 @@ Since Pulsar SQL is powered by [Trino (formerly Presto SQL)](https://trino.io), :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.4.0/sql-overview.md b/site2/website/versioned_docs/version-2.4.0/sql-overview.md index 2f827f4267933..5b01dfde56e7f 100644 --- a/site2/website/versioned_docs/version-2.4.0/sql-overview.md +++ b/site2/website/versioned_docs/version-2.4.0/sql-overview.md @@ -4,7 +4,7 @@ title: Pulsar SQL Overview sidebar_label: "Overview" --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL)](https://trino.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL.md)](https://trino.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.4.0/standalone-docker.md b/site2/website/versioned_docs/version-2.4.0/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.4.0/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.4.0/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.4.0/standalone.md b/site2/website/versioned_docs/version-2.4.0/standalone.md index 16f3f2c6b796d..77c4bb7151c24 100644 --- a/site2/website/versioned_docs/version-2.4.0/standalone.md +++ b/site2/website/versioned_docs/version-2.4.0/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -51,9 +51,9 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](reference-pulsar-admin). +`bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](reference-pulsar-admin.md). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -62,7 +62,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -111,7 +111,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -125,7 +125,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -158,12 +158,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: diff --git a/site2/website/versioned_docs/version-2.4.1/admin-api-clusters.md b/site2/website/versioned_docs/version-2.4.1/admin-api-clusters.md index 04db2f60759bc..1255ad25d7aac 100644 --- a/site2/website/versioned_docs/version-2.4.1/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.4.1/admin-api-clusters.md @@ -88,7 +88,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -112,11 +112,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block > - For the latest and complete information about `Java admin API`, including classes, methods, descriptions, and more, see [Java admin API doc](https://pulsar.apache.org/api/admin/). -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -121,7 +121,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.4.1/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.4.1/client-libraries-cpp.md index 5ba9e03d89521..b709463ff0309 100644 --- a/site2/website/versioned_docs/version-2.4.1/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.4.1/client-libraries-cpp.md @@ -661,7 +661,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema This section describes some examples about schema. For more information about -schema, see [Pulsar schema](schema-get-started). +schema, see [Pulsar schema](schema-get-started.md). ### Avro schema diff --git a/site2/website/versioned_docs/version-2.4.1/client-libraries-go.md b/site2/website/versioned_docs/version-2.4.1/client-libraries-go.md index 33059163f4c6b..d3c2fa5d2fc85 100644 --- a/site2/website/versioned_docs/version-2.4.1/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.4.1/client-libraries-go.md @@ -55,7 +55,7 @@ $ ./test_example ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -81,7 +81,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -1022,7 +1022,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker @@ -1042,7 +1042,7 @@ opts := pulsar.ClientOptions{ ## OAuth2 authentication -To use [OAuth2 authentication](security-oauth2), you'll need to configure your client to perform the following operations. +To use [OAuth2 authentication](security-oauth2.md), you'll need to configure your client to perform the following operations. This example shows how to configure OAuth2 authentication. ```go diff --git a/site2/website/versioned_docs/version-2.4.1/client-libraries-java.md b/site2/website/versioned_docs/version-2.4.1/client-libraries-java.md index 686bd1241dfb5..1e526941f766c 100644 --- a/site2/website/versioned_docs/version-2.4.1/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.4.1/client-libraries-java.md @@ -5,16 +5,16 @@ sidebar_label: "Java" original_id: client-libraries-java --- -The Pulsar Java client can be used both to create Java producers, consumers, and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +The Pulsar Java client can be used both to create Java producers, consumers, and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. Javadoc for the Pulsar client is divided up into two domains, by package: Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) -This document will focus only on the client API for producing and consuming messages on Pulsar topics. For a guide to using the Java admin client, see [The Pulsar admin interface](admin-api-overview). +This document will focus only on the client API for producing and consuming messages on Pulsar topics. For a guide to using the Java admin client, see [The Pulsar admin interface](admin-api-overview.md). ## Installation @@ -54,7 +54,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -80,7 +80,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -111,7 +111,7 @@ PulsarClient client = PulsarClient.builder() ``` > #### Default broker URLs for standalone clusters -> If you're running a cluster in [standalone mode](getting-started-standalone), the broker will be available at the `pulsar://localhost:6650` URL by default. +> If you're running a cluster in [standalone mode](getting-started-standalone.md), the broker will be available at the `pulsar://localhost:6650` URL by default. If you create a client, you may use the `loadConf` configuration. Below are the available parameters used in `loadConf`. @@ -214,7 +214,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -630,7 +630,7 @@ The code sample above shows pointing the `Reader` object to a specific message ( ## Schemas -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. Here's an example: +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. Here's an example: ```java @@ -744,7 +744,7 @@ The following schema formats are currently available for Java: ## Authentication -Pulsar currently supports two authentication schemes: [TLS](security-tls-authentication.md) and [Athenz](security-athenz). The Pulsar Java client can be used with both. +Pulsar currently supports two authentication schemes: [TLS](security-tls-authentication.md) and [Athenz](security-athenz.md). The Pulsar Java client can be used with both. ### TLS Authentication @@ -771,7 +771,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` diff --git a/site2/website/versioned_docs/version-2.4.1/client-libraries-python.md b/site2/website/versioned_docs/version-2.4.1/client-libraries-python.md index 1c22f7240d1e0..3f52095b3c4bc 100644 --- a/site2/website/versioned_docs/version-2.4.1/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.4.1/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -The Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +The Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. ## Installation diff --git a/site2/website/versioned_docs/version-2.4.1/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.4.1/client-libraries-websocket.md index a6e60361749de..0b31ffe78161a 100644 --- a/site2/website/versioned_docs/version-2.4.1/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.4.1/client-libraries-websocket.md @@ -4,14 +4,14 @@ title: Pulsar WebSocket API sidebar_label: "WebSocket" --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.4.1/client-libraries.md b/site2/website/versioned_docs/version-2.4.1/client-libraries.md index 536cd0ccda82f..8f91dfbbbdd8c 100644 --- a/site2/website/versioned_docs/version-2.4.1/client-libraries.md +++ b/site2/website/versioned_docs/version-2.4.1/client-libraries.md @@ -8,12 +8,12 @@ Pulsar supports the following client libraries: |Language|Documentation|Release note|Code repo |---|---|---|--- -Java |- [User doc](client-libraries-java)

    - [API doc](https://pulsar.apache.org/api/client/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client) -C++ | - [User doc](client-libraries-cpp)

    - [API doc](https://pulsar.apache.org/api/cpp/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp) -Python | - [User doc](client-libraries-python)

    - [API doc](https://pulsar.apache.org/api/python/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) -WebSocket| [User doc](client-libraries-websocket) | [Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-websocket) +Java |- [User doc](client-libraries-java.md)

    - [API doc](https://pulsar.apache.org/api/client/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client) +C++ | - [User doc](client-libraries-cpp.md)

    - [API doc](https://pulsar.apache.org/api/cpp/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp) +Python | - [User doc](client-libraries-python.md)

    - [API doc](https://pulsar.apache.org/api/python/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) +WebSocket| [User doc](client-libraries-websocket.md) | [Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-websocket) Go client|[User doc](client-libraries-go.md)|[Here](https://github.com/apache/pulsar-client-go/blob/master/CHANGELOG) |[Here](https://github.com/apache/pulsar-client-go) -Node.js|[User doc](client-libraries-node)|[Here](https://github.com/apache/pulsar-client-node/releases) |[Here](https://github.com/apache/pulsar-client-node) +Node.js|[User doc](client-libraries-node.md)|[Here](https://github.com/apache/pulsar-client-node/releases) |[Here](https://github.com/apache/pulsar-client-node) C# |[User doc](client-libraries-dotnet.md)| [Here](https://github.com/apache/pulsar-dotpulsar/blob/master/CHANGELOG)|[Here](https://github.com/apache/pulsar-dotpulsar) :::note diff --git a/site2/website/versioned_docs/version-2.4.1/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.4.1/concepts-architecture-overview.md index 0d96b59f8ae91..1ce346fc16941 100644 --- a/site2/website/versioned_docs/version-2.4.1/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.4.1/concepts-architecture-overview.md @@ -4,7 +4,7 @@ title: Architecture Overview sidebar_label: "Architecture" --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -16,20 +16,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers. The producers connect to the brokers to publish messages and the consumers connect to the brokers to consume the messages. -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -39,9 +39,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -139,17 +139,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -157,7 +157,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.4.1/concepts-authentication.md b/site2/website/versioned_docs/version-2.4.1/concepts-authentication.md index 335da8dc5a519..fbefead69f382 100644 --- a/site2/website/versioned_docs/version-2.4.1/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.4.1/concepts-authentication.md @@ -4,5 +4,5 @@ title: Authentication and Authorization sidebar_label: "Authentication and Authorization" --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.4.1/concepts-clients.md b/site2/website/versioned_docs/version-2.4.1/concepts-clients.md index 65201b5ce1cd2..c0fe61cd06e76 100644 --- a/site2/website/versioned_docs/version-2.4.1/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.4.1/concepts-clients.md @@ -4,12 +4,12 @@ title: Pulsar Clients sidebar_label: "Clients" --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -37,7 +37,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.4.1/concepts-messaging.md b/site2/website/versioned_docs/version-2.4.1/concepts-messaging.md index f55e8cfe42fa9..4a50484c79e0e 100644 --- a/site2/website/versioned_docs/version-2.4.1/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.4.1/concepts-messaging.md @@ -24,7 +24,7 @@ Publish time | The timestamp of when the message was published (automatically ap Event time | An optional timestamp that applications can attach to the message representing when something happened, e.g. when the message was processed. The event time of a message is 0 if none is explicitly set. -> For a more in-depth breakdown of Pulsar message contents, see the documentation on Pulsar's [binary protocol](developing-binary-protocol). +> For a more in-depth breakdown of Pulsar message contents, see the documentation on Pulsar's [binary protocol](developing-binary-protocol.md). ## Producers @@ -67,7 +67,7 @@ Messages can be received from [brokers](reference-terminology.md#broker) either ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -294,7 +294,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -306,7 +306,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -344,7 +344,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -399,7 +399,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -422,12 +422,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, which means that you don't need to modify your Pulsar client code. Instead, you only need to make administrative changes (see the [Managing message deduplication](cookbooks-deduplication) cookbook for a guide). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, which means that you don't need to modify your Pulsar client code. Instead, you only need to make administrative changes (see the [Managing message deduplication](cookbooks-deduplication.md) cookbook for a guide). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.4.1/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.4.1/concepts-multi-tenancy.md index 8a17e72a4c8b3..26bd6a2fd183f 100644 --- a/site2/website/versioned_docs/version-2.4.1/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.4.1/concepts-multi-tenancy.md @@ -4,7 +4,7 @@ title: Multi Tenancy sidebar_label: "Multi Tenancy" --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -20,7 +20,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -28,7 +28,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.4.1/concepts-overview.md b/site2/website/versioned_docs/version-2.4.1/concepts-overview.md index 119d46508ed28..bf4993a3110b5 100644 --- a/site2/website/versioned_docs/version-2.4.1/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.4.1/concepts-overview.md @@ -8,15 +8,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.4.1/concepts-replication.md b/site2/website/versioned_docs/version-2.4.1/concepts-replication.md index d4653ca7aecd8..9cc3a7c3415bd 100644 --- a/site2/website/versioned_docs/version-2.4.1/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.4.1/concepts-replication.md @@ -8,7 +8,7 @@ Regardless of industries, when an unforeseen event occurs and brings day-to-day Pulsar's geo-replication mechanism is typically used for disaster recovery, enabling the replication of persistently stored message data across multiple data centers. For instance, your application is publishing data in one region and you would like to process it for consumption in other regions. With Pulsar’s geo-replication mechanism, messages can be produced and consumed in different geo-locations. -The diagram below illustrates the process of [geo-replication](administration-geo). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. +The diagram below illustrates the process of [geo-replication](administration-geo.md). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. ![A typical geo-replication example with full-mesh pattern](/assets/full-mesh-replication.svg) diff --git a/site2/website/versioned_docs/version-2.4.1/concepts-schema-registry.md b/site2/website/versioned_docs/version-2.4.1/concepts-schema-registry.md index 72cb46662cbb2..c39f384601410 100644 --- a/site2/website/versioned_docs/version-2.4.1/concepts-schema-registry.md +++ b/site2/website/versioned_docs/version-2.4.1/concepts-schema-registry.md @@ -17,7 +17,7 @@ Both approaches are available in Pulsar, and you're free to adopt one or the oth #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-go.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-go.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ## Basic architecture @@ -47,7 +47,7 @@ Schemas are versioned in succession.The schema is stored in a broker that handle Once a version is assigned/fetched to/for a schema, all subsequent messages produced by that producer are tagged with the appropriate version. -In order to illustrate how schema versioning works, let's walk through an example. Imagine that the Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begin sending messages: +In order to illustrate how schema versioning works, let's walk through an example. Imagine that the Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begin sending messages: ```java diff --git a/site2/website/versioned_docs/version-2.4.1/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.4.1/concepts-tiered-storage.md index e912d77834c33..cfdd19dff58c9 100644 --- a/site2/website/versioned_docs/version-2.4.1/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.4.1/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3 and Google Cloud Storage (GCS) for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.4.1/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.4.1/concepts-topic-compaction.md index 790ce76b7c5e0..300cf68956dd1 100644 --- a/site2/website/versioned_docs/version-2.4.1/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.4.1/concepts-topic-compaction.md @@ -6,7 +6,7 @@ sidebar_label: "Topic Compaction" Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -23,7 +23,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.4.1/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.4.1/cookbooks-compaction.md index fca6caf0355ea..2d6432726f476 100644 --- a/site2/website/versioned_docs/version-2.4.1/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.4.1/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually {#trigger} -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.4.1/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.4.1/cookbooks-non-persistent.md index d40c4fbd0cf01..1346d19a04263 100644 --- a/site2/website/versioned_docs/version-2.4.1/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.4.1/cookbooks-non-persistent.md @@ -40,7 +40,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -54,7 +54,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.4.1/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.4.1/cookbooks-partitioned.md index 25896934788fc..c857d019edc02 100644 --- a/site2/website/versioned_docs/version-2.4.1/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.4.1/cookbooks-partitioned.md @@ -3,4 +3,4 @@ id: cookbooks-partitioned title: Partitioned topics sidebar_label: "Partitioned Topics" --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.4.1/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.4.1/cookbooks-retention-expiry.md index 86160c3f7ecf8..cd764fad6eb37 100644 --- a/site2/website/versioned_docs/version-2.4.1/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.4.1/cookbooks-retention-expiry.md @@ -21,7 +21,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.4.1/deploy-aws.md b/site2/website/versioned_docs/version-2.4.1/deploy-aws.md index 5db846fe790f2..60ac4c22d7672 100644 --- a/site2/website/versioned_docs/version-2.4.1/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.4.1/deploy-aws.md @@ -4,7 +4,7 @@ title: Deploying a Pulsar cluster on AWS using Terraform and Ansible sidebar_label: "Amazon Web Services" --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -209,7 +209,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.4.1/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.4.1/deploy-bare-metal-multi-cluster.md index 70dff45080b03..8a765d8843f43 100644 --- a/site2/website/versioned_docs/version-2.4.1/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.4.1/deploy-bare-metal-multi-cluster.md @@ -6,13 +6,13 @@ sidebar_label: "Bare metal multi-cluster" :::tip -1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster). -2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster.md). +2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo).Deploying a multi-cluster Pulsar instance consists of the following steps: +A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md).Deploying a multi-cluster Pulsar instance consists of the following steps: 1. Deploying two separate ZooKeeper quorums: a local quorum for each cluster in the instance and a configuration store quorum for instance-wide tasks 2. Initializing cluster metadata for each cluster @@ -21,7 +21,7 @@ A Pulsar instance consists of multiple Pulsar clusters working in unison. You ca > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. ## System requirement @@ -65,7 +65,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -244,7 +244,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -363,16 +363,16 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. -You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > **Service discovery already provided by many scheduling systems** -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.4.1/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.4.1/deploy-bare-metal.md index 0da5f4d027dae..0f9e17adacef2 100644 --- a/site2/website/versioned_docs/version-2.4.1/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.4.1/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you're interested in experimenting with Pulsar or using it in a startup or on a single team, we recommend opting for a single cluster. If you do need to run a multi-cluster Pulsar instance, -however, see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +however, see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and make sure it is installed under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and make sure it is installed under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you could reference this [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you could reference this [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -266,9 +266,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (we don't recommend using a different port). -`--web-service-url-tls` | If you're using [TLS](security-tls-transport), you'll also need to specify a TLS web service URL for the cluster. The default port is 8443 (we don't recommend using a different port). +`--web-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster. The default port is 8443 (we don't recommend using a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (we don't recommend using a different port). -`--broker-service-url-tls` | If you're using [TLS](security-tls-transport), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (we don't recommend using a different port). +`--broker-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (we don't recommend using a different port). ## Deploying a BookKeeper cluster @@ -378,7 +378,7 @@ webServicePortTls=8443 ### Enabling Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -396,7 +396,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you would like to learn more options about deploying functions worker, please checkout [Deploy and manage functions worker](functions-worker). +If you would like to learn more options about deploying functions worker, please checkout [Deploy and manage functions worker](functions-worker.md). ### Starting Brokers diff --git a/site2/website/versioned_docs/version-2.4.1/deploy-dcos.md b/site2/website/versioned_docs/version-2.4.1/deploy-dcos.md index 07f446edaffd6..599d3a1b0454c 100644 --- a/site2/website/versioned_docs/version-2.4.1/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.4.1/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -To enable all built-in [Pulsar IO](io-overview) connectors in your Pulsar deploymente, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). +To enable all built-in [Pulsar IO](io-overview.md) connectors in your Pulsar deploymente, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.4.1/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.4.1/deploy-kubernetes.md index 4e170dcd4711d..7455247c901d4 100644 --- a/site2/website/versioned_docs/version-2.4.1/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.4.1/deploy-kubernetes.md @@ -5,6 +5,6 @@ sidebar_label: "Kubernetes" --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.4.1/deploy-monitoring.md b/site2/website/versioned_docs/version-2.4.1/deploy-monitoring.md index adf3587b7aeb3..d581fe2edfacb 100644 --- a/site2/website/versioned_docs/version-2.4.1/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.4.1/deploy-monitoring.md @@ -112,7 +112,7 @@ http://$FUNCTIONS_WORKER_ADDRESS:$WORKER_PORT/metrics: You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy). +When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy.md). ## Dashboards @@ -120,7 +120,7 @@ When you collect time series statistics, the major problem is to make sure the n ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager). +The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.4.1/develop-schema.md b/site2/website/versioned_docs/version-2.4.1/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.4.1/develop-schema.md +++ b/site2/website/versioned_docs/version-2.4.1/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.4.1/functions-deploy.md b/site2/website/versioned_docs/version-2.4.1/functions-deploy.md index 51f1140693dbe..d9496b3ed5b48 100644 --- a/site2/website/versioned_docs/version-2.4.1/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.4.1/functions-deploy.md @@ -9,12 +9,12 @@ original_id: functions-deploy To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface diff --git a/site2/website/versioned_docs/version-2.4.1/functions-develop.md b/site2/website/versioned_docs/version-2.4.1/functions-develop.md index ac397835ba1a9..ef4929f0f8a3f 100644 --- a/site2/website/versioned_docs/version-2.4.1/functions-develop.md +++ b/site2/website/versioned_docs/version-2.4.1/functions-develop.md @@ -837,7 +837,7 @@ public class MetricRecorderFunction implements Function { ``` -> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring) guide. +> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring.md) guide. @@ -863,7 +863,7 @@ class MetricRecorderFunction(Function): ```` ### Access metrics -To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring) in Pulsar. +To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring.md) in Pulsar. ## State storage Pulsar Functions use [Apache BookKeeper](https://bookkeeper.apache.org) as a state storage interface. Pulsar installation, including the local standalone installation, includes deployment of BookKeeper bookies. diff --git a/site2/website/versioned_docs/version-2.4.1/functions-overview.md b/site2/website/versioned_docs/version-2.4.1/functions-overview.md index f222cbed63c90..b08da31097daf 100644 --- a/site2/website/versioned_docs/version-2.4.1/functions-overview.md +++ b/site2/website/versioned_docs/version-2.4.1/functions-overview.md @@ -150,7 +150,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.4.1/functions-worker.md b/site2/website/versioned_docs/version-2.4.1/functions-worker.md index 643d9cabeee84..e4af5d0defe46 100644 --- a/site2/website/versioned_docs/version-2.4.1/functions-worker.md +++ b/site2/website/versioned_docs/version-2.4.1/functions-worker.md @@ -5,7 +5,7 @@ sidebar_label: "Setup: Pulsar Functions Worker" original_id: functions-worker --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either of the two options based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -124,7 +124,7 @@ tlsTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -151,7 +151,7 @@ properties: ``` For *Token Authentication* prodivder, add necessary settings under `properties` if needed. -See [Token Authentication](security-token-admin) for more details. +See [Token Authentication](security-token-admin.md) for more details. ``` diff --git a/site2/website/versioned_docs/version-2.4.1/getting-started-docker.md b/site2/website/versioned_docs/version-2.4.1/getting-started-docker.md index 26908245cb2c9..904d67401668a 100644 --- a/site2/website/versioned_docs/version-2.4.1/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.4.1/getting-started-docker.md @@ -43,8 +43,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -103,7 +103,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.4.1/getting-started-standalone.md b/site2/website/versioned_docs/version-2.4.1/getting-started-standalone.md index 3b4f8b065abd5..a5099d29359dc 100644 --- a/site2/website/versioned_docs/version-2.4.1/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.4.1/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -57,9 +57,9 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](reference-pulsar-admin). +`bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](reference-pulsar-admin.md). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.4.1/io-cdc.md b/site2/website/versioned_docs/version-2.4.1/io-cdc.md index df373977e3eaa..a3527dc0a9d65 100644 --- a/site2/website/versioned_docs/version-2.4.1/io-cdc.md +++ b/site2/website/versioned_docs/version-2.4.1/io-cdc.md @@ -12,8 +12,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.4.1/io-develop.md b/site2/website/versioned_docs/version-2.4.1/io-develop.md index 6ff3742f04176..86e2dd961c873 100644 --- a/site2/website/versioned_docs/version-2.4.1/io-develop.md +++ b/site2/website/versioned_docs/version-2.4.1/io-develop.md @@ -8,12 +8,12 @@ original_id: io-develop This guide describes how developers can write new connectors for Pulsar IO to move data between Pulsar and other systems. It describes how to create a Pulsar IO connector. -Pulsar IO connectors are specialized [Pulsar Functions](functions-overview). So writing +Pulsar IO connectors are specialized [Pulsar Functions](functions-overview.md). So writing a Pulsar IO connector is as simple as writing a Pulsar function. Pulsar IO connectors come in two flavors: {@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}, which import data from another system, and {@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}, -which export data to another system. For example, [KinesisSink](io-kinesis) would export -the messages of a Pulsar topic to a Kinesis stream, and [RabbitmqSource](io-rabbitmq) would import +which export data to another system. For example, [KinesisSink](io-kinesis.md) would export +the messages of a Pulsar topic to a Kinesis stream, and [RabbitmqSource](io-rabbitmq.md) would import the messages of a RabbitMQ queue to a Pulsar topic. ### Developing @@ -137,7 +137,7 @@ separate integration tests to verify end-to-end functionality. In Pulsar, we are ### Packaging Once you've developed and tested your connector, you must package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. There are two approaches described +to a [Pulsar Functions](functions-overview.md) cluster. There are two approaches described here work with Pulsar Functions' runtime. If you plan to package and distribute your connector for others to use, you are obligated to @@ -153,7 +153,7 @@ The easiest approach to packaging a Pulsar IO connector is to create a NAR packa NAR stands for NiFi Archive. It is a custom packaging mechanism used by Apache NiFi, to provide a bit of Java ClassLoader isolation. For more details, you can read this [blog post](https://medium.com/hashmapinc/nifi-nar-files-explained-14113f7796fd) to understand -how NAR works. Pulsar uses the same mechanism for packaging all the [builtin connectors](io-connectors). +how NAR works. Pulsar uses the same mechanism for packaging all the [builtin connectors](io-connectors.md). All what you need is to include this [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin) in your maven project for your connector. For example: diff --git a/site2/website/versioned_docs/version-2.4.1/io-managing.md b/site2/website/versioned_docs/version-2.4.1/io-managing.md index cce795bbae86a..8c834410a5d0f 100644 --- a/site2/website/versioned_docs/version-2.4.1/io-managing.md +++ b/site2/website/versioned_docs/version-2.4.1/io-managing.md @@ -50,7 +50,7 @@ For details, consult the documentation for [individual connectors](io-overview.m ## Running Connectors -Pulsar connectors can be managed using the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands of the [`pulsar-admin`](reference-pulsar-admin) CLI tool. +Pulsar connectors can be managed using the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands of the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. ### Running sources @@ -166,8 +166,8 @@ Here's an example to submit a Cassandra sink: ## Monitoring Connectors -Since Pulsar IO connectors are running as [Pulsar Functions](functions-overview), so you can use [`functions`](reference-pulsar-admin.md#source) commands -available in the [`pulsar-admin`](reference-pulsar-admin) CLI tool. +Since Pulsar IO connectors are running as [Pulsar Functions](functions-overview.md), so you can use [`functions`](reference-pulsar-admin.md#source) commands +available in the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. ### Retrieve Connector Metadata diff --git a/site2/website/versioned_docs/version-2.4.1/io-overview.md b/site2/website/versioned_docs/version-2.4.1/io-overview.md index 2a6ecaee5883f..ea49e4b042e8c 100644 --- a/site2/website/versioned_docs/version-2.4.1/io-overview.md +++ b/site2/website/versioned_docs/version-2.4.1/io-overview.md @@ -157,7 +157,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.4.1/io-quickstart.md b/site2/website/versioned_docs/version-2.4.1/io-quickstart.md index 10a03809e1749..256171b47c0d1 100644 --- a/site2/website/versioned_docs/version-2.4.1/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.4.1/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you will be able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you will be able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However all the commands used in this tutorial should be able to be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -214,11 +214,11 @@ configs: ``` -To learn more about Cassandra Connector, see [Cassandra Connector](io-cassandra). +To learn more about Cassandra Connector, see [Cassandra Connector](io-cassandra.md). ### Submit a Cassandra sink -Pulsar provides the [CLI](reference-cli-tools) for running and managing Pulsar I/O connectors. +Pulsar provides the [CLI](reference-cli-tools.md) for running and managing Pulsar I/O connectors. We can run following command to sink a sink connector with type `cassandra` and config file `examples/cassandra-sink.yml`. @@ -582,7 +582,7 @@ Now that we have a MySQL running locally. In this section, we will configure a J ### Submit a JDBC sink -Pulsar provides the [CLI](admin-api-overview) for running and managing Pulsar I/O connectors. +Pulsar provides the [CLI](admin-api-overview.md) for running and managing Pulsar I/O connectors. This example creates a sink connector and specifies the desired information. diff --git a/site2/website/versioned_docs/version-2.4.1/io-use.md b/site2/website/versioned_docs/version-2.4.1/io-use.md index fc46136d56b01..1e9a635a79643 100644 --- a/site2/website/versioned_docs/version-2.4.1/io-use.md +++ b/site2/website/versioned_docs/version-2.4.1/io-use.md @@ -15,7 +15,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.4.1/pulsar-2.0.md b/site2/website/versioned_docs/version-2.4.1/pulsar-2.0.md index 560c8c1bd7cb7..07fba365e9ff2 100644 --- a/site2/website/versioned_docs/version-2.4.1/pulsar-2.0.md +++ b/site2/website/versioned_docs/version-2.4.1/pulsar-2.0.md @@ -4,13 +4,13 @@ title: Pulsar 2.0 sidebar_label: "Pulsar 2.0" --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.4.1/reference-cli-tools.md b/site2/website/versioned_docs/version-2.4.1/reference-cli-tools.md index 1c3d601e00c88..dcac11835f4b9 100644 --- a/site2/website/versioned_docs/version-2.4.1/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.4.1/reference-cli-tools.md @@ -6,7 +6,7 @@ sidebar_label: "Pulsar CLI tools" Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) @@ -825,7 +825,7 @@ Options ### `transaction` -Run a transaction. For more information, see [Pulsar transactions](txn-why). +Run a transaction. For more information, see [Pulsar transactions](txn-why.md). **Usage** diff --git a/site2/website/versioned_docs/version-2.4.1/schema-manage.md b/site2/website/versioned_docs/version-2.4.1/schema-manage.md index 9cd7f29c5456c..f0199571010ac 100644 --- a/site2/website/versioned_docs/version-2.4.1/schema-manage.md +++ b/site2/website/versioned_docs/version-2.4.1/schema-manage.md @@ -135,7 +135,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.4.1/schema-understand.md b/site2/website/versioned_docs/version-2.4.1/schema-understand.md index 7214ca0b378c4..a22afc7e8faee 100644 --- a/site2/website/versioned_docs/version-2.4.1/schema-understand.md +++ b/site2/website/versioned_docs/version-2.4.1/schema-understand.md @@ -373,7 +373,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```text diff --git a/site2/website/versioned_docs/version-2.4.1/security-athenz.md b/site2/website/versioned_docs/version-2.4.1/security-athenz.md index 947c3f470be46..8a39fe25316d0 100644 --- a/site2/website/versioned_docs/version-2.4.1/security-athenz.md +++ b/site2/website/versioned_docs/version-2.4.1/security-athenz.md @@ -76,7 +76,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.4.1/security-authorization.md b/site2/website/versioned_docs/version-2.4.1/security-authorization.md index 28200ad0744b9..ef78adc6cad0c 100644 --- a/site2/website/versioned_docs/version-2.4.1/security-authorization.md +++ b/site2/website/versioned_docs/version-2.4.1/security-authorization.md @@ -27,7 +27,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you can not only use superuser roles for administrators and clients but also for broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you can not only use superuser roles for administrators and clients but also for broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. The broker still checks the authorization of the request when the broker receives the forwarded request. @@ -58,7 +58,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -86,7 +86,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication diff --git a/site2/website/versioned_docs/version-2.4.1/security-jwt.md b/site2/website/versioned_docs/version-2.4.1/security-jwt.md index 4160d16d0e224..8ab3d1b11660c 100644 --- a/site2/website/versioned_docs/version-2.4.1/security-jwt.md +++ b/site2/website/versioned_docs/version-2.4.1/security-jwt.md @@ -31,11 +31,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -302,7 +302,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.4.1/security-kerberos.md b/site2/website/versioned_docs/version-2.4.1/security-kerberos.md index 670586a9d1691..c49fa3bea1fce 100644 --- a/site2/website/versioned_docs/version-2.4.1/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.4.1/security-kerberos.md @@ -373,7 +373,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.4.1/security-overview.md b/site2/website/versioned_docs/version-2.4.1/security-overview.md index f9a623334eb27..ee7d8f9b60b5d 100644 --- a/site2/website/versioned_docs/version-2.4.1/security-overview.md +++ b/site2/website/versioned_docs/version-2.4.1/security-overview.md @@ -17,7 +17,7 @@ You'd better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent one or more clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses the [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses the [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication Providers @@ -30,7 +30,7 @@ Currently Pulsar supports the following authentication providers: ## Contents -- [Encryption](security-tls-transport.md) and [Authentication](security-tls-authentication) using TLS +- [Encryption](security-tls-transport.md) and [Authentication](security-tls-authentication.md) using TLS - [Authentication using Athenz](security-athenz.md) - [Authentication using Kerberos](security-kerberos.md) - [Authorization and ACLs](security-authorization.md) diff --git a/site2/website/versioned_docs/version-2.4.1/security-tls-authentication.md b/site2/website/versioned_docs/version-2.4.1/security-tls-authentication.md index c8ad87971776d..5339637f6982e 100644 --- a/site2/website/versioned_docs/version-2.4.1/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.4.1/security-tls-authentication.md @@ -7,7 +7,7 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. ### Create client certificates @@ -99,7 +99,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -119,7 +119,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.4.1/security-tls-transport.md b/site2/website/versioned_docs/version-2.4.1/security-tls-transport.md index 91e1cc67a1a32..2a80e5254a613 100644 --- a/site2/website/versioned_docs/version-2.4.1/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.4.1/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,7 +27,7 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). ## Create TLS certificates @@ -128,7 +128,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Broker Configuration -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -199,7 +199,7 @@ The examples below show hostname verification being disabled for the Java client ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.4.1/security-token-admin.md b/site2/website/versioned_docs/version-2.4.1/security-token-admin.md index 4accb85a75ae0..a4370fb3f8190 100644 --- a/site2/website/versioned_docs/version-2.4.1/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.4.1/security-token-admin.md @@ -162,7 +162,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.4.1/security-token-client.md b/site2/website/versioned_docs/version-2.4.1/security-token-client.md index b955bcddacf7d..2ef64e045e3b7 100644 --- a/site2/website/versioned_docs/version-2.4.1/security-token-client.md +++ b/site2/website/versioned_docs/version-2.4.1/security-token-client.md @@ -24,12 +24,12 @@ eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJKb2UifQ.ipevRNuRP6HflG8cFKnmUPtypruRC4fb1DWtoLL6 Application specifies the token when you are creating the client instance. An alternative is to pass a "token supplier" (a function that returns the token when the client library needs one). -See [Token authentication admin](security-token-admin) for a reference on how to enable token +See [Token authentication admin](security-token-admin.md) for a reference on how to enable token authentication on a Pulsar cluster. ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.4.1/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.4.1/sql-deployment-configurations.md index 639ac988b3409..069c6ac9c92da 100644 --- a/site2/website/versioned_docs/version-2.4.1/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.4.1/sql-deployment-configurations.md @@ -141,7 +141,7 @@ Since Pulsar SQL is powered by [Trino (formerly Presto SQL)](https://trino.io), :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.4.1/sql-overview.md b/site2/website/versioned_docs/version-2.4.1/sql-overview.md index 2f827f4267933..5b01dfde56e7f 100644 --- a/site2/website/versioned_docs/version-2.4.1/sql-overview.md +++ b/site2/website/versioned_docs/version-2.4.1/sql-overview.md @@ -4,7 +4,7 @@ title: Pulsar SQL Overview sidebar_label: "Overview" --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL)](https://trino.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL.md)](https://trino.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.4.1/standalone-docker.md b/site2/website/versioned_docs/version-2.4.1/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.4.1/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.4.1/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.4.1/standalone.md b/site2/website/versioned_docs/version-2.4.1/standalone.md index fb510051b1728..c6058908889af 100644 --- a/site2/website/versioned_docs/version-2.4.1/standalone.md +++ b/site2/website/versioned_docs/version-2.4.1/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -57,9 +57,9 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](reference-pulsar-admin). +`bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](reference-pulsar-admin.md). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.4.2/admin-api-clusters.md b/site2/website/versioned_docs/version-2.4.2/admin-api-clusters.md index 04db2f60759bc..1255ad25d7aac 100644 --- a/site2/website/versioned_docs/version-2.4.2/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.4.2/admin-api-clusters.md @@ -88,7 +88,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -112,11 +112,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block > - For the latest and complete information about `Java admin API`, including classes, methods, descriptions, and more, see [Java admin API doc](https://pulsar.apache.org/api/admin/). -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -121,7 +121,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.4.2/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.4.2/client-libraries-cpp.md index 5ba9e03d89521..b709463ff0309 100644 --- a/site2/website/versioned_docs/version-2.4.2/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.4.2/client-libraries-cpp.md @@ -661,7 +661,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema This section describes some examples about schema. For more information about -schema, see [Pulsar schema](schema-get-started). +schema, see [Pulsar schema](schema-get-started.md). ### Avro schema diff --git a/site2/website/versioned_docs/version-2.4.2/client-libraries-go.md b/site2/website/versioned_docs/version-2.4.2/client-libraries-go.md index 33059163f4c6b..d3c2fa5d2fc85 100644 --- a/site2/website/versioned_docs/version-2.4.2/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.4.2/client-libraries-go.md @@ -55,7 +55,7 @@ $ ./test_example ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -81,7 +81,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -1022,7 +1022,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker @@ -1042,7 +1042,7 @@ opts := pulsar.ClientOptions{ ## OAuth2 authentication -To use [OAuth2 authentication](security-oauth2), you'll need to configure your client to perform the following operations. +To use [OAuth2 authentication](security-oauth2.md), you'll need to configure your client to perform the following operations. This example shows how to configure OAuth2 authentication. ```go diff --git a/site2/website/versioned_docs/version-2.4.2/client-libraries-java.md b/site2/website/versioned_docs/version-2.4.2/client-libraries-java.md index 686bd1241dfb5..1e526941f766c 100644 --- a/site2/website/versioned_docs/version-2.4.2/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.4.2/client-libraries-java.md @@ -5,16 +5,16 @@ sidebar_label: "Java" original_id: client-libraries-java --- -The Pulsar Java client can be used both to create Java producers, consumers, and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +The Pulsar Java client can be used both to create Java producers, consumers, and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. Javadoc for the Pulsar client is divided up into two domains, by package: Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) -This document will focus only on the client API for producing and consuming messages on Pulsar topics. For a guide to using the Java admin client, see [The Pulsar admin interface](admin-api-overview). +This document will focus only on the client API for producing and consuming messages on Pulsar topics. For a guide to using the Java admin client, see [The Pulsar admin interface](admin-api-overview.md). ## Installation @@ -54,7 +54,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -80,7 +80,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -111,7 +111,7 @@ PulsarClient client = PulsarClient.builder() ``` > #### Default broker URLs for standalone clusters -> If you're running a cluster in [standalone mode](getting-started-standalone), the broker will be available at the `pulsar://localhost:6650` URL by default. +> If you're running a cluster in [standalone mode](getting-started-standalone.md), the broker will be available at the `pulsar://localhost:6650` URL by default. If you create a client, you may use the `loadConf` configuration. Below are the available parameters used in `loadConf`. @@ -214,7 +214,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -630,7 +630,7 @@ The code sample above shows pointing the `Reader` object to a specific message ( ## Schemas -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. Here's an example: +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. Here's an example: ```java @@ -744,7 +744,7 @@ The following schema formats are currently available for Java: ## Authentication -Pulsar currently supports two authentication schemes: [TLS](security-tls-authentication.md) and [Athenz](security-athenz). The Pulsar Java client can be used with both. +Pulsar currently supports two authentication schemes: [TLS](security-tls-authentication.md) and [Athenz](security-athenz.md). The Pulsar Java client can be used with both. ### TLS Authentication @@ -771,7 +771,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` diff --git a/site2/website/versioned_docs/version-2.4.2/client-libraries-python.md b/site2/website/versioned_docs/version-2.4.2/client-libraries-python.md index 1c22f7240d1e0..3f52095b3c4bc 100644 --- a/site2/website/versioned_docs/version-2.4.2/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.4.2/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -The Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +The Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. ## Installation diff --git a/site2/website/versioned_docs/version-2.4.2/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.4.2/client-libraries-websocket.md index a6e60361749de..0b31ffe78161a 100644 --- a/site2/website/versioned_docs/version-2.4.2/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.4.2/client-libraries-websocket.md @@ -4,14 +4,14 @@ title: Pulsar WebSocket API sidebar_label: "WebSocket" --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.4.2/client-libraries.md b/site2/website/versioned_docs/version-2.4.2/client-libraries.md index 536cd0ccda82f..8f91dfbbbdd8c 100644 --- a/site2/website/versioned_docs/version-2.4.2/client-libraries.md +++ b/site2/website/versioned_docs/version-2.4.2/client-libraries.md @@ -8,12 +8,12 @@ Pulsar supports the following client libraries: |Language|Documentation|Release note|Code repo |---|---|---|--- -Java |- [User doc](client-libraries-java)

    - [API doc](https://pulsar.apache.org/api/client/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client) -C++ | - [User doc](client-libraries-cpp)

    - [API doc](https://pulsar.apache.org/api/cpp/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp) -Python | - [User doc](client-libraries-python)

    - [API doc](https://pulsar.apache.org/api/python/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) -WebSocket| [User doc](client-libraries-websocket) | [Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-websocket) +Java |- [User doc](client-libraries-java.md)

    - [API doc](https://pulsar.apache.org/api/client/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client) +C++ | - [User doc](client-libraries-cpp.md)

    - [API doc](https://pulsar.apache.org/api/cpp/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp) +Python | - [User doc](client-libraries-python.md)

    - [API doc](https://pulsar.apache.org/api/python/)|[Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) +WebSocket| [User doc](client-libraries-websocket.md) | [Here](https://pulsar.apache.org/release-notes/)|[Here](https://github.com/apache/pulsar/tree/master/pulsar-websocket) Go client|[User doc](client-libraries-go.md)|[Here](https://github.com/apache/pulsar-client-go/blob/master/CHANGELOG) |[Here](https://github.com/apache/pulsar-client-go) -Node.js|[User doc](client-libraries-node)|[Here](https://github.com/apache/pulsar-client-node/releases) |[Here](https://github.com/apache/pulsar-client-node) +Node.js|[User doc](client-libraries-node.md)|[Here](https://github.com/apache/pulsar-client-node/releases) |[Here](https://github.com/apache/pulsar-client-node) C# |[User doc](client-libraries-dotnet.md)| [Here](https://github.com/apache/pulsar-dotpulsar/blob/master/CHANGELOG)|[Here](https://github.com/apache/pulsar-dotpulsar) :::note diff --git a/site2/website/versioned_docs/version-2.4.2/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.4.2/concepts-architecture-overview.md index 0d96b59f8ae91..1ce346fc16941 100644 --- a/site2/website/versioned_docs/version-2.4.2/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.4.2/concepts-architecture-overview.md @@ -4,7 +4,7 @@ title: Architecture Overview sidebar_label: "Architecture" --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -16,20 +16,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers. The producers connect to the brokers to publish messages and the consumers connect to the brokers to consume the messages. -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -39,9 +39,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -139,17 +139,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -157,7 +157,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.4.2/concepts-authentication.md b/site2/website/versioned_docs/version-2.4.2/concepts-authentication.md index 335da8dc5a519..fbefead69f382 100644 --- a/site2/website/versioned_docs/version-2.4.2/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.4.2/concepts-authentication.md @@ -4,5 +4,5 @@ title: Authentication and Authorization sidebar_label: "Authentication and Authorization" --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.4.2/concepts-clients.md b/site2/website/versioned_docs/version-2.4.2/concepts-clients.md index 65201b5ce1cd2..c0fe61cd06e76 100644 --- a/site2/website/versioned_docs/version-2.4.2/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.4.2/concepts-clients.md @@ -4,12 +4,12 @@ title: Pulsar Clients sidebar_label: "Clients" --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -37,7 +37,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.4.2/concepts-messaging.md b/site2/website/versioned_docs/version-2.4.2/concepts-messaging.md index f55e8cfe42fa9..4a50484c79e0e 100644 --- a/site2/website/versioned_docs/version-2.4.2/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.4.2/concepts-messaging.md @@ -24,7 +24,7 @@ Publish time | The timestamp of when the message was published (automatically ap Event time | An optional timestamp that applications can attach to the message representing when something happened, e.g. when the message was processed. The event time of a message is 0 if none is explicitly set. -> For a more in-depth breakdown of Pulsar message contents, see the documentation on Pulsar's [binary protocol](developing-binary-protocol). +> For a more in-depth breakdown of Pulsar message contents, see the documentation on Pulsar's [binary protocol](developing-binary-protocol.md). ## Producers @@ -67,7 +67,7 @@ Messages can be received from [brokers](reference-terminology.md#broker) either ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -294,7 +294,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -306,7 +306,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -344,7 +344,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -399,7 +399,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -422,12 +422,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, which means that you don't need to modify your Pulsar client code. Instead, you only need to make administrative changes (see the [Managing message deduplication](cookbooks-deduplication) cookbook for a guide). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, which means that you don't need to modify your Pulsar client code. Instead, you only need to make administrative changes (see the [Managing message deduplication](cookbooks-deduplication.md) cookbook for a guide). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.4.2/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.4.2/concepts-multi-tenancy.md index 8a17e72a4c8b3..26bd6a2fd183f 100644 --- a/site2/website/versioned_docs/version-2.4.2/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.4.2/concepts-multi-tenancy.md @@ -4,7 +4,7 @@ title: Multi Tenancy sidebar_label: "Multi Tenancy" --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -20,7 +20,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -28,7 +28,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.4.2/concepts-overview.md b/site2/website/versioned_docs/version-2.4.2/concepts-overview.md index 119d46508ed28..bf4993a3110b5 100644 --- a/site2/website/versioned_docs/version-2.4.2/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.4.2/concepts-overview.md @@ -8,15 +8,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.4.2/concepts-replication.md b/site2/website/versioned_docs/version-2.4.2/concepts-replication.md index d4653ca7aecd8..9cc3a7c3415bd 100644 --- a/site2/website/versioned_docs/version-2.4.2/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.4.2/concepts-replication.md @@ -8,7 +8,7 @@ Regardless of industries, when an unforeseen event occurs and brings day-to-day Pulsar's geo-replication mechanism is typically used for disaster recovery, enabling the replication of persistently stored message data across multiple data centers. For instance, your application is publishing data in one region and you would like to process it for consumption in other regions. With Pulsar’s geo-replication mechanism, messages can be produced and consumed in different geo-locations. -The diagram below illustrates the process of [geo-replication](administration-geo). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. +The diagram below illustrates the process of [geo-replication](administration-geo.md). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. ![A typical geo-replication example with full-mesh pattern](/assets/full-mesh-replication.svg) diff --git a/site2/website/versioned_docs/version-2.4.2/concepts-schema-registry.md b/site2/website/versioned_docs/version-2.4.2/concepts-schema-registry.md index 4d7fd42f6375a..628a17728737b 100644 --- a/site2/website/versioned_docs/version-2.4.2/concepts-schema-registry.md +++ b/site2/website/versioned_docs/version-2.4.2/concepts-schema-registry.md @@ -17,7 +17,7 @@ Both approaches are available in Pulsar, and you're free to adopt one or the oth #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-go.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-go.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ## Basic architecture @@ -47,7 +47,7 @@ Schemas are versioned in succession.The schema is stored in a broker that handle Once a version is assigned/fetched to/for a schema, all subsequent messages produced by that producer are tagged with the appropriate version. -In order to illustrate how schema versioning works, let's walk through an example. Imagine that the Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begin sending messages: +In order to illustrate how schema versioning works, let's walk through an example. Imagine that the Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begin sending messages: ```java diff --git a/site2/website/versioned_docs/version-2.4.2/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.4.2/concepts-tiered-storage.md index e912d77834c33..cfdd19dff58c9 100644 --- a/site2/website/versioned_docs/version-2.4.2/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.4.2/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3 and Google Cloud Storage (GCS) for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.4.2/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.4.2/concepts-topic-compaction.md index 790ce76b7c5e0..300cf68956dd1 100644 --- a/site2/website/versioned_docs/version-2.4.2/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.4.2/concepts-topic-compaction.md @@ -6,7 +6,7 @@ sidebar_label: "Topic Compaction" Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -23,7 +23,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.4.2/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.4.2/cookbooks-compaction.md index fca6caf0355ea..2d6432726f476 100644 --- a/site2/website/versioned_docs/version-2.4.2/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.4.2/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually {#trigger} -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.4.2/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.4.2/cookbooks-non-persistent.md index d40c4fbd0cf01..1346d19a04263 100644 --- a/site2/website/versioned_docs/version-2.4.2/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.4.2/cookbooks-non-persistent.md @@ -40,7 +40,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -54,7 +54,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.4.2/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.4.2/cookbooks-partitioned.md index 25896934788fc..c857d019edc02 100644 --- a/site2/website/versioned_docs/version-2.4.2/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.4.2/cookbooks-partitioned.md @@ -3,4 +3,4 @@ id: cookbooks-partitioned title: Partitioned topics sidebar_label: "Partitioned Topics" --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.4.2/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.4.2/cookbooks-retention-expiry.md index 86160c3f7ecf8..cd764fad6eb37 100644 --- a/site2/website/versioned_docs/version-2.4.2/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.4.2/cookbooks-retention-expiry.md @@ -21,7 +21,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.4.2/deploy-aws.md b/site2/website/versioned_docs/version-2.4.2/deploy-aws.md index 5db846fe790f2..60ac4c22d7672 100644 --- a/site2/website/versioned_docs/version-2.4.2/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.4.2/deploy-aws.md @@ -4,7 +4,7 @@ title: Deploying a Pulsar cluster on AWS using Terraform and Ansible sidebar_label: "Amazon Web Services" --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -209,7 +209,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.4.2/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.4.2/deploy-bare-metal-multi-cluster.md index 70dff45080b03..8a765d8843f43 100644 --- a/site2/website/versioned_docs/version-2.4.2/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.4.2/deploy-bare-metal-multi-cluster.md @@ -6,13 +6,13 @@ sidebar_label: "Bare metal multi-cluster" :::tip -1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster). -2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster.md). +2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo).Deploying a multi-cluster Pulsar instance consists of the following steps: +A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md).Deploying a multi-cluster Pulsar instance consists of the following steps: 1. Deploying two separate ZooKeeper quorums: a local quorum for each cluster in the instance and a configuration store quorum for instance-wide tasks 2. Initializing cluster metadata for each cluster @@ -21,7 +21,7 @@ A Pulsar instance consists of multiple Pulsar clusters working in unison. You ca > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. ## System requirement @@ -65,7 +65,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -244,7 +244,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -363,16 +363,16 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. -You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > **Service discovery already provided by many scheduling systems** -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.4.2/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.4.2/deploy-bare-metal.md index 0da5f4d027dae..0f9e17adacef2 100644 --- a/site2/website/versioned_docs/version-2.4.2/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.4.2/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you're interested in experimenting with Pulsar or using it in a startup or on a single team, we recommend opting for a single cluster. If you do need to run a multi-cluster Pulsar instance, -however, see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +however, see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and make sure it is installed under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and make sure it is installed under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you could reference this [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you could reference this [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -266,9 +266,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (we don't recommend using a different port). -`--web-service-url-tls` | If you're using [TLS](security-tls-transport), you'll also need to specify a TLS web service URL for the cluster. The default port is 8443 (we don't recommend using a different port). +`--web-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster. The default port is 8443 (we don't recommend using a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (we don't recommend using a different port). -`--broker-service-url-tls` | If you're using [TLS](security-tls-transport), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (we don't recommend using a different port). +`--broker-service-url-tls` | If you're using [TLS](security-tls-transport.md), you'll also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (we don't recommend using a different port). ## Deploying a BookKeeper cluster @@ -378,7 +378,7 @@ webServicePortTls=8443 ### Enabling Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -396,7 +396,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you would like to learn more options about deploying functions worker, please checkout [Deploy and manage functions worker](functions-worker). +If you would like to learn more options about deploying functions worker, please checkout [Deploy and manage functions worker](functions-worker.md). ### Starting Brokers diff --git a/site2/website/versioned_docs/version-2.4.2/deploy-dcos.md b/site2/website/versioned_docs/version-2.4.2/deploy-dcos.md index 07f446edaffd6..599d3a1b0454c 100644 --- a/site2/website/versioned_docs/version-2.4.2/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.4.2/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -To enable all built-in [Pulsar IO](io-overview) connectors in your Pulsar deploymente, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). +To enable all built-in [Pulsar IO](io-overview.md) connectors in your Pulsar deploymente, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.4.2/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.4.2/deploy-kubernetes.md index 4e170dcd4711d..7455247c901d4 100644 --- a/site2/website/versioned_docs/version-2.4.2/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.4.2/deploy-kubernetes.md @@ -5,6 +5,6 @@ sidebar_label: "Kubernetes" --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.4.2/deploy-monitoring.md b/site2/website/versioned_docs/version-2.4.2/deploy-monitoring.md index adf3587b7aeb3..d581fe2edfacb 100644 --- a/site2/website/versioned_docs/version-2.4.2/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.4.2/deploy-monitoring.md @@ -112,7 +112,7 @@ http://$FUNCTIONS_WORKER_ADDRESS:$WORKER_PORT/metrics: You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy). +When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy.md). ## Dashboards @@ -120,7 +120,7 @@ When you collect time series statistics, the major problem is to make sure the n ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager). +The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.4.2/develop-schema.md b/site2/website/versioned_docs/version-2.4.2/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.4.2/develop-schema.md +++ b/site2/website/versioned_docs/version-2.4.2/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.4.2/functions-deploy.md b/site2/website/versioned_docs/version-2.4.2/functions-deploy.md index 51f1140693dbe..d9496b3ed5b48 100644 --- a/site2/website/versioned_docs/version-2.4.2/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.4.2/functions-deploy.md @@ -9,12 +9,12 @@ original_id: functions-deploy To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface diff --git a/site2/website/versioned_docs/version-2.4.2/functions-develop.md b/site2/website/versioned_docs/version-2.4.2/functions-develop.md index f9f60b74468d5..1f0a0ac10da81 100644 --- a/site2/website/versioned_docs/version-2.4.2/functions-develop.md +++ b/site2/website/versioned_docs/version-2.4.2/functions-develop.md @@ -837,7 +837,7 @@ public class MetricRecorderFunction implements Function { ``` -> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring) guide. +> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring.md) guide.     @@ -863,7 +863,7 @@ class MetricRecorderFunction(Function): ```` ### Access metrics -To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring) in Pulsar. +To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring.md) in Pulsar. ## State storage Pulsar Functions use [Apache BookKeeper](https://bookkeeper.apache.org) as a state storage interface. Pulsar installation, including the local standalone installation, includes deployment of BookKeeper bookies. diff --git a/site2/website/versioned_docs/version-2.4.2/functions-overview.md b/site2/website/versioned_docs/version-2.4.2/functions-overview.md index f222cbed63c90..b08da31097daf 100644 --- a/site2/website/versioned_docs/version-2.4.2/functions-overview.md +++ b/site2/website/versioned_docs/version-2.4.2/functions-overview.md @@ -150,7 +150,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.4.2/functions-worker.md b/site2/website/versioned_docs/version-2.4.2/functions-worker.md index 643d9cabeee84..e4af5d0defe46 100644 --- a/site2/website/versioned_docs/version-2.4.2/functions-worker.md +++ b/site2/website/versioned_docs/version-2.4.2/functions-worker.md @@ -5,7 +5,7 @@ sidebar_label: "Setup: Pulsar Functions Worker" original_id: functions-worker --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either of the two options based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -124,7 +124,7 @@ tlsTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -151,7 +151,7 @@ properties: ``` For *Token Authentication* prodivder, add necessary settings under `properties` if needed. -See [Token Authentication](security-token-admin) for more details. +See [Token Authentication](security-token-admin.md) for more details. ``` diff --git a/site2/website/versioned_docs/version-2.4.2/getting-started-docker.md b/site2/website/versioned_docs/version-2.4.2/getting-started-docker.md index 26908245cb2c9..904d67401668a 100644 --- a/site2/website/versioned_docs/version-2.4.2/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.4.2/getting-started-docker.md @@ -43,8 +43,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -103,7 +103,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.4.2/getting-started-standalone.md b/site2/website/versioned_docs/version-2.4.2/getting-started-standalone.md index 5d6cc59ca5081..d9ad00ba2a250 100644 --- a/site2/website/versioned_docs/version-2.4.2/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.4.2/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -57,9 +57,9 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](reference-pulsar-admin). +`bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](reference-pulsar-admin.md). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -203,7 +203,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.4.2/io-cdc.md b/site2/website/versioned_docs/version-2.4.2/io-cdc.md index df373977e3eaa..a3527dc0a9d65 100644 --- a/site2/website/versioned_docs/version-2.4.2/io-cdc.md +++ b/site2/website/versioned_docs/version-2.4.2/io-cdc.md @@ -12,8 +12,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.4.2/io-develop.md b/site2/website/versioned_docs/version-2.4.2/io-develop.md index 6ff3742f04176..86e2dd961c873 100644 --- a/site2/website/versioned_docs/version-2.4.2/io-develop.md +++ b/site2/website/versioned_docs/version-2.4.2/io-develop.md @@ -8,12 +8,12 @@ original_id: io-develop This guide describes how developers can write new connectors for Pulsar IO to move data between Pulsar and other systems. It describes how to create a Pulsar IO connector. -Pulsar IO connectors are specialized [Pulsar Functions](functions-overview). So writing +Pulsar IO connectors are specialized [Pulsar Functions](functions-overview.md). So writing a Pulsar IO connector is as simple as writing a Pulsar function. Pulsar IO connectors come in two flavors: {@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}, which import data from another system, and {@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}, -which export data to another system. For example, [KinesisSink](io-kinesis) would export -the messages of a Pulsar topic to a Kinesis stream, and [RabbitmqSource](io-rabbitmq) would import +which export data to another system. For example, [KinesisSink](io-kinesis.md) would export +the messages of a Pulsar topic to a Kinesis stream, and [RabbitmqSource](io-rabbitmq.md) would import the messages of a RabbitMQ queue to a Pulsar topic. ### Developing @@ -137,7 +137,7 @@ separate integration tests to verify end-to-end functionality. In Pulsar, we are ### Packaging Once you've developed and tested your connector, you must package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. There are two approaches described +to a [Pulsar Functions](functions-overview.md) cluster. There are two approaches described here work with Pulsar Functions' runtime. If you plan to package and distribute your connector for others to use, you are obligated to @@ -153,7 +153,7 @@ The easiest approach to packaging a Pulsar IO connector is to create a NAR packa NAR stands for NiFi Archive. It is a custom packaging mechanism used by Apache NiFi, to provide a bit of Java ClassLoader isolation. For more details, you can read this [blog post](https://medium.com/hashmapinc/nifi-nar-files-explained-14113f7796fd) to understand -how NAR works. Pulsar uses the same mechanism for packaging all the [builtin connectors](io-connectors). +how NAR works. Pulsar uses the same mechanism for packaging all the [builtin connectors](io-connectors.md). All what you need is to include this [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin) in your maven project for your connector. For example: diff --git a/site2/website/versioned_docs/version-2.4.2/io-managing.md b/site2/website/versioned_docs/version-2.4.2/io-managing.md index cce795bbae86a..8c834410a5d0f 100644 --- a/site2/website/versioned_docs/version-2.4.2/io-managing.md +++ b/site2/website/versioned_docs/version-2.4.2/io-managing.md @@ -50,7 +50,7 @@ For details, consult the documentation for [individual connectors](io-overview.m ## Running Connectors -Pulsar connectors can be managed using the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands of the [`pulsar-admin`](reference-pulsar-admin) CLI tool. +Pulsar connectors can be managed using the [`source`](reference-pulsar-admin.md#source) and [`sink`](reference-pulsar-admin.md#sink) commands of the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. ### Running sources @@ -166,8 +166,8 @@ Here's an example to submit a Cassandra sink: ## Monitoring Connectors -Since Pulsar IO connectors are running as [Pulsar Functions](functions-overview), so you can use [`functions`](reference-pulsar-admin.md#source) commands -available in the [`pulsar-admin`](reference-pulsar-admin) CLI tool. +Since Pulsar IO connectors are running as [Pulsar Functions](functions-overview.md), so you can use [`functions`](reference-pulsar-admin.md#source) commands +available in the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. ### Retrieve Connector Metadata diff --git a/site2/website/versioned_docs/version-2.4.2/io-overview.md b/site2/website/versioned_docs/version-2.4.2/io-overview.md index 2a6ecaee5883f..ea49e4b042e8c 100644 --- a/site2/website/versioned_docs/version-2.4.2/io-overview.md +++ b/site2/website/versioned_docs/version-2.4.2/io-overview.md @@ -157,7 +157,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.4.2/io-quickstart.md b/site2/website/versioned_docs/version-2.4.2/io-quickstart.md index 10a03809e1749..256171b47c0d1 100644 --- a/site2/website/versioned_docs/version-2.4.2/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.4.2/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you will be able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you will be able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However all the commands used in this tutorial should be able to be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -214,11 +214,11 @@ configs: ``` -To learn more about Cassandra Connector, see [Cassandra Connector](io-cassandra). +To learn more about Cassandra Connector, see [Cassandra Connector](io-cassandra.md). ### Submit a Cassandra sink -Pulsar provides the [CLI](reference-cli-tools) for running and managing Pulsar I/O connectors. +Pulsar provides the [CLI](reference-cli-tools.md) for running and managing Pulsar I/O connectors. We can run following command to sink a sink connector with type `cassandra` and config file `examples/cassandra-sink.yml`. @@ -582,7 +582,7 @@ Now that we have a MySQL running locally. In this section, we will configure a J ### Submit a JDBC sink -Pulsar provides the [CLI](admin-api-overview) for running and managing Pulsar I/O connectors. +Pulsar provides the [CLI](admin-api-overview.md) for running and managing Pulsar I/O connectors. This example creates a sink connector and specifies the desired information. diff --git a/site2/website/versioned_docs/version-2.4.2/io-use.md b/site2/website/versioned_docs/version-2.4.2/io-use.md index 61cef063e673f..acd9fa8258d7d 100644 --- a/site2/website/versioned_docs/version-2.4.2/io-use.md +++ b/site2/website/versioned_docs/version-2.4.2/io-use.md @@ -15,7 +15,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.4.2/pulsar-2.0.md b/site2/website/versioned_docs/version-2.4.2/pulsar-2.0.md index 560c8c1bd7cb7..07fba365e9ff2 100644 --- a/site2/website/versioned_docs/version-2.4.2/pulsar-2.0.md +++ b/site2/website/versioned_docs/version-2.4.2/pulsar-2.0.md @@ -4,13 +4,13 @@ title: Pulsar 2.0 sidebar_label: "Pulsar 2.0" --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.4.2/reference-cli-tools.md b/site2/website/versioned_docs/version-2.4.2/reference-cli-tools.md index 1c3d601e00c88..dcac11835f4b9 100644 --- a/site2/website/versioned_docs/version-2.4.2/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.4.2/reference-cli-tools.md @@ -6,7 +6,7 @@ sidebar_label: "Pulsar CLI tools" Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) @@ -825,7 +825,7 @@ Options ### `transaction` -Run a transaction. For more information, see [Pulsar transactions](txn-why). +Run a transaction. For more information, see [Pulsar transactions](txn-why.md). **Usage** diff --git a/site2/website/versioned_docs/version-2.4.2/schema-manage.md b/site2/website/versioned_docs/version-2.4.2/schema-manage.md index 9cd7f29c5456c..f0199571010ac 100644 --- a/site2/website/versioned_docs/version-2.4.2/schema-manage.md +++ b/site2/website/versioned_docs/version-2.4.2/schema-manage.md @@ -135,7 +135,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.4.2/schema-understand.md b/site2/website/versioned_docs/version-2.4.2/schema-understand.md index c137de712379a..11b4836aa88dd 100644 --- a/site2/website/versioned_docs/version-2.4.2/schema-understand.md +++ b/site2/website/versioned_docs/version-2.4.2/schema-understand.md @@ -373,7 +373,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```text diff --git a/site2/website/versioned_docs/version-2.4.2/security-athenz.md b/site2/website/versioned_docs/version-2.4.2/security-athenz.md index 947c3f470be46..8a39fe25316d0 100644 --- a/site2/website/versioned_docs/version-2.4.2/security-athenz.md +++ b/site2/website/versioned_docs/version-2.4.2/security-athenz.md @@ -76,7 +76,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.4.2/security-authorization.md b/site2/website/versioned_docs/version-2.4.2/security-authorization.md index 28200ad0744b9..ef78adc6cad0c 100644 --- a/site2/website/versioned_docs/version-2.4.2/security-authorization.md +++ b/site2/website/versioned_docs/version-2.4.2/security-authorization.md @@ -27,7 +27,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you can not only use superuser roles for administrators and clients but also for broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you can not only use superuser roles for administrators and clients but also for broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. The broker still checks the authorization of the request when the broker receives the forwarded request. @@ -58,7 +58,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -86,7 +86,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication diff --git a/site2/website/versioned_docs/version-2.4.2/security-jwt.md b/site2/website/versioned_docs/version-2.4.2/security-jwt.md index 4160d16d0e224..8ab3d1b11660c 100644 --- a/site2/website/versioned_docs/version-2.4.2/security-jwt.md +++ b/site2/website/versioned_docs/version-2.4.2/security-jwt.md @@ -31,11 +31,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -302,7 +302,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.4.2/security-kerberos.md b/site2/website/versioned_docs/version-2.4.2/security-kerberos.md index 670586a9d1691..c49fa3bea1fce 100644 --- a/site2/website/versioned_docs/version-2.4.2/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.4.2/security-kerberos.md @@ -373,7 +373,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.4.2/security-overview.md b/site2/website/versioned_docs/version-2.4.2/security-overview.md index f9a623334eb27..ee7d8f9b60b5d 100644 --- a/site2/website/versioned_docs/version-2.4.2/security-overview.md +++ b/site2/website/versioned_docs/version-2.4.2/security-overview.md @@ -17,7 +17,7 @@ You'd better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent one or more clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses the [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses the [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication Providers @@ -30,7 +30,7 @@ Currently Pulsar supports the following authentication providers: ## Contents -- [Encryption](security-tls-transport.md) and [Authentication](security-tls-authentication) using TLS +- [Encryption](security-tls-transport.md) and [Authentication](security-tls-authentication.md) using TLS - [Authentication using Athenz](security-athenz.md) - [Authentication using Kerberos](security-kerberos.md) - [Authorization and ACLs](security-authorization.md) diff --git a/site2/website/versioned_docs/version-2.4.2/security-tls-authentication.md b/site2/website/versioned_docs/version-2.4.2/security-tls-authentication.md index c8ad87971776d..5339637f6982e 100644 --- a/site2/website/versioned_docs/version-2.4.2/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.4.2/security-tls-authentication.md @@ -7,7 +7,7 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. ### Create client certificates @@ -99,7 +99,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -119,7 +119,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.4.2/security-tls-transport.md b/site2/website/versioned_docs/version-2.4.2/security-tls-transport.md index 91e1cc67a1a32..2a80e5254a613 100644 --- a/site2/website/versioned_docs/version-2.4.2/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.4.2/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,7 +27,7 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). ## Create TLS certificates @@ -128,7 +128,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Broker Configuration -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -199,7 +199,7 @@ The examples below show hostname verification being disabled for the Java client ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.4.2/security-token-admin.md b/site2/website/versioned_docs/version-2.4.2/security-token-admin.md index 4accb85a75ae0..a4370fb3f8190 100644 --- a/site2/website/versioned_docs/version-2.4.2/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.4.2/security-token-admin.md @@ -162,7 +162,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.4.2/security-token-client.md b/site2/website/versioned_docs/version-2.4.2/security-token-client.md index b955bcddacf7d..2ef64e045e3b7 100644 --- a/site2/website/versioned_docs/version-2.4.2/security-token-client.md +++ b/site2/website/versioned_docs/version-2.4.2/security-token-client.md @@ -24,12 +24,12 @@ eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJKb2UifQ.ipevRNuRP6HflG8cFKnmUPtypruRC4fb1DWtoLL6 Application specifies the token when you are creating the client instance. An alternative is to pass a "token supplier" (a function that returns the token when the client library needs one). -See [Token authentication admin](security-token-admin) for a reference on how to enable token +See [Token authentication admin](security-token-admin.md) for a reference on how to enable token authentication on a Pulsar cluster. ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.4.2/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.4.2/sql-deployment-configurations.md index 639ac988b3409..069c6ac9c92da 100644 --- a/site2/website/versioned_docs/version-2.4.2/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.4.2/sql-deployment-configurations.md @@ -141,7 +141,7 @@ Since Pulsar SQL is powered by [Trino (formerly Presto SQL)](https://trino.io), :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.4.2/sql-overview.md b/site2/website/versioned_docs/version-2.4.2/sql-overview.md index 2f827f4267933..5b01dfde56e7f 100644 --- a/site2/website/versioned_docs/version-2.4.2/sql-overview.md +++ b/site2/website/versioned_docs/version-2.4.2/sql-overview.md @@ -4,7 +4,7 @@ title: Pulsar SQL Overview sidebar_label: "Overview" --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL)](https://trino.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL.md)](https://trino.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.4.2/standalone-docker.md b/site2/website/versioned_docs/version-2.4.2/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.4.2/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.4.2/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.4.2/standalone.md b/site2/website/versioned_docs/version-2.4.2/standalone.md index 6de554859a639..ad9dccb0ec023 100644 --- a/site2/website/versioned_docs/version-2.4.2/standalone.md +++ b/site2/website/versioned_docs/version-2.4.2/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -57,9 +57,9 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](reference-pulsar-admin). +`bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](reference-pulsar-admin.md). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -203,7 +203,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.5.0/admin-api-clusters.md b/site2/website/versioned_docs/version-2.5.0/admin-api-clusters.md index 04db2f60759bc..1255ad25d7aac 100644 --- a/site2/website/versioned_docs/version-2.5.0/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.5.0/admin-api-clusters.md @@ -88,7 +88,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -112,11 +112,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block > - For the latest and complete information about `Java admin API`, including classes, methods, descriptions, and more, see [Java admin API doc](https://pulsar.apache.org/api/admin/). -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -121,7 +121,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block For a guide to managing message persistence, retention, and expiry in Pulsar, see [this cookbook](cookbooks-retention-expiry). +> For a guide to managing message persistence, retention, and expiry in Pulsar, see [this cookbook](cookbooks-retention-expiry.md). ### Deploy BookKeeper diff --git a/site2/website/versioned_docs/version-2.5.0/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.5.0/client-libraries-cpp.md index c5522ac126197..bbc74dbb31eab 100644 --- a/site2/website/versioned_docs/version-2.5.0/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.5.0/client-libraries-cpp.md @@ -207,7 +207,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema -This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started). +This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started.md). ### Create producer with Avro schema diff --git a/site2/website/versioned_docs/version-2.5.0/client-libraries-go.md b/site2/website/versioned_docs/version-2.5.0/client-libraries-go.md index fa8c26f5d09e5..e09d7bd7ae15f 100644 --- a/site2/website/versioned_docs/version-2.5.0/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.5.0/client-libraries-go.md @@ -16,7 +16,7 @@ The Pulsar Go client can be used to create Pulsar [producers](#producers), [cons ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Installing go package @@ -50,7 +50,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -68,7 +68,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -229,14 +229,14 @@ Parameter | Description | Default `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | `Properties`| Attach a set of application defined properties to the producer. This properties will be visible in the topic stats | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | Set the number of max pending messages across all the partitions. This setting will be used to lower the max pending messages for each partition `MaxPendingMessages(int)`, if the total exceeds the configured value.| `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | `Batching` | Control whether automatic batching of messages is enabled for the producer. | false `BatchingMaxPublishDelay` | Set the time period within which the messages sent will be batched (default: 1ms) if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or until | 1ms `BatchingMaxMessages` | Set the maximum number of messages permitted in a batch. (default: 1000) If set to a value greater than 1, messages will be queued until this threshold is reached or batch interval has elapsed | 1000 @@ -505,7 +505,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.5.0/client-libraries-java.md b/site2/website/versioned_docs/version-2.5.0/client-libraries-java.md index d5dc98fcb997f..34764c35540a5 100644 --- a/site2/website/versioned_docs/version-2.5.0/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.5.0/client-libraries-java.md @@ -5,16 +5,16 @@ sidebar_label: "Java" original_id: client-libraries-java --- -You can use Pulsar Java client to create Java producers, consumers, and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +You can use Pulsar Java client to create Java producers, consumers, and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -54,7 +54,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -80,7 +80,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -111,7 +111,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -234,7 +234,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -757,7 +757,7 @@ Total hash range size is 65536, so the max end of the range should be less than ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -871,7 +871,7 @@ The following schema formats are currently available for Java: ## Authentication -Pulsar currently supports two authentication schemes: [TLS](security-tls-authentication.md) and [Athenz](security-athenz). You can use the Pulsar Java client with both. +Pulsar currently supports two authentication schemes: [TLS](security-tls-authentication.md) and [Athenz](security-athenz.md). You can use the Pulsar Java client with both. ### TLS Authentication @@ -898,7 +898,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` diff --git a/site2/website/versioned_docs/version-2.5.0/client-libraries-node.md b/site2/website/versioned_docs/version-2.5.0/client-libraries-node.md index b4fca51d9db31..592dfa75c2c42 100644 --- a/site2/website/versioned_docs/version-2.5.0/client-libraries-node.md +++ b/site2/website/versioned_docs/version-2.5.0/client-libraries-node.md @@ -42,7 +42,7 @@ Also, this library works only in Node.js 10.x or later because it uses the [`nod ::: ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here is an example for `localhost`: @@ -60,7 +60,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication), the URL will look like something like this: +If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication.md), the URL will look like something like this: ```http @@ -95,7 +95,7 @@ The following configurable parameters are available for Pulsar clients: | Parameter | Description | Default | | :-------- | :---------- | :------ | | `serviceUrl` | The connection URL for the Pulsar cluster. See [above](#connection-urls) for more info. | | -| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication) for more info. | | +| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication.md) for more info. | | | `operationTimeoutSeconds` | The timeout for Node.js client operations (creating producers, subscribing to and unsubscribing from [topics](reference-terminology.md#topic)). Retries will occur until this threshold is reached, at which point the operation will fail. | 30 | | `ioThreads` | The number of threads to use for handling connections to Pulsar [brokers](reference-terminology.md#broker). | 1 | | `messageListenerThreads` | The number of threads used by message listeners ([consumers](#consumers) and [readers](#readers)). | 1 | @@ -145,7 +145,7 @@ Pulsar Node.js producers have the following methods available: | :-------- | :---------- | :------ | | `topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer publishes messages. The topic format is `` or `//`. For example, `sample/ns1/my-topic`. | | | `producerName` | A name for the producer. If you do not explicitly assign a name, Pulsar will automatically generate a globally unique name. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | | -| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30000 | +| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30000 | | `initialSequenceId` | The initial sequence ID of the message. When producer send message, add sequence ID to message. The ID is increased each time to send. | | | `maxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `send` method will fail *unless* `blockIfQueueFull` is set to `true`. | 1000 | | `maxPendingMessagesAcrossPartitions` | The maximum size of the sum of partition's pending queue. | 50000 | diff --git a/site2/website/versioned_docs/version-2.5.0/client-libraries-python.md b/site2/website/versioned_docs/version-2.5.0/client-libraries-python.md index cbb6018b25bf2..d804344562f5f 100644 --- a/site2/website/versioned_docs/version-2.5.0/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.5.0/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -The Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +The Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. ## Installation diff --git a/site2/website/versioned_docs/version-2.5.0/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.5.0/client-libraries-websocket.md index a6e60361749de..0b31ffe78161a 100644 --- a/site2/website/versioned_docs/version-2.5.0/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.5.0/client-libraries-websocket.md @@ -4,14 +4,14 @@ title: Pulsar WebSocket API sidebar_label: "WebSocket" --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.5.0/client-libraries.md b/site2/website/versioned_docs/version-2.5.0/client-libraries.md index 357db9b158c30..7bb8611208d5c 100644 --- a/site2/website/versioned_docs/version-2.5.0/client-libraries.md +++ b/site2/website/versioned_docs/version-2.5.0/client-libraries.md @@ -14,30 +14,30 @@ Pulsar supports the following client libraries: ## Java client -For instructions on how to use the Pulsar Java client to produce and consume messages, see [Pulsar Java client](client-libraries-java). +For instructions on how to use the Pulsar Java client to produce and consume messages, see [Pulsar Java client](client-libraries-java.md). Two independent sets of Javadoc API docs are available. Library | Purpose :-------|:------- -[`org.apache.pulsar.client.api`](/api/client) | The [Pulsar Java client](client-libraries-java) is used to produce and consume messages on Pulsar topics. -[`org.apache.pulsar.client.admin`](/api/admin) | The Java client for the [Pulsar admin interface](admin-api-overview). +[`org.apache.pulsar.client.api`](/api/client) | The [Pulsar Java client](client-libraries-java.md) is used to produce and consume messages on Pulsar topics. +[`org.apache.pulsar.client.admin`](/api/admin) | The Java client for the [Pulsar admin interface](admin-api-overview.md). ## Go client -For a tutorial on using the Pulsar Go client, see [Pulsar Go client](client-libraries-go). +For a tutorial on using the Pulsar Go client, see [Pulsar Go client](client-libraries-go.md). ## Python client -For a tutorial on using the Pulsar Python client, see [Pulsar Python client](client-libraries-python). +For a tutorial on using the Pulsar Python client, see [Pulsar Python client](client-libraries-python.md). There are also [pdoc](https://github.com/BurntSushi/pdoc)-generated API docs for the Python client [here](/api/python). ## C++ client -For a tutorial on using the Pulsar C++ clent, see [Pulsar C++ client](client-libraries-cpp). +For a tutorial on using the Pulsar C++ clent, see [Pulsar C++ client](client-libraries-cpp.md). There are also [Doxygen](http://www.doxygen.nl/)-generated API docs for the C++ client [here](/api/cpp). diff --git a/site2/website/versioned_docs/version-2.5.0/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.5.0/concepts-architecture-overview.md index 0d96b59f8ae91..1ce346fc16941 100644 --- a/site2/website/versioned_docs/version-2.5.0/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.5.0/concepts-architecture-overview.md @@ -4,7 +4,7 @@ title: Architecture Overview sidebar_label: "Architecture" --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -16,20 +16,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers. The producers connect to the brokers to publish messages and the consumers connect to the brokers to consume the messages. -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -39,9 +39,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -139,17 +139,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -157,7 +157,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.5.0/concepts-authentication.md b/site2/website/versioned_docs/version-2.5.0/concepts-authentication.md index 335da8dc5a519..fbefead69f382 100644 --- a/site2/website/versioned_docs/version-2.5.0/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.5.0/concepts-authentication.md @@ -4,5 +4,5 @@ title: Authentication and Authorization sidebar_label: "Authentication and Authorization" --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.5.0/concepts-clients.md b/site2/website/versioned_docs/version-2.5.0/concepts-clients.md index 72342f3bdd749..41f423263208b 100644 --- a/site2/website/versioned_docs/version-2.5.0/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.5.0/concepts-clients.md @@ -5,7 +5,7 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. diff --git a/site2/website/versioned_docs/version-2.5.0/concepts-messaging.md b/site2/website/versioned_docs/version-2.5.0/concepts-messaging.md index a7727a91a66e6..0cd176b1cdd79 100644 --- a/site2/website/versioned_docs/version-2.5.0/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.5.0/concepts-messaging.md @@ -24,7 +24,7 @@ Publish time | The timestamp of when the message was published (automatically ap Event time | An optional timestamp that applications can attach to the message representing when something happened, e.g. when the message was processed. The event time of a message is 0 if none is explicitly set. -> For a more in-depth breakdown of Pulsar message contents, see the documentation on Pulsar's [binary protocol](developing-binary-protocol). +> For a more in-depth breakdown of Pulsar message contents, see the documentation on Pulsar's [binary protocol](developing-binary-protocol.md). ## Producers @@ -67,7 +67,7 @@ Messages can be received from [brokers](reference-terminology.md#broker) either ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -298,7 +298,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -310,7 +310,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -348,7 +348,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -403,7 +403,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -426,12 +426,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, which means that you don't need to modify your Pulsar client code. Instead, you only need to make administrative changes (see the [Managing message deduplication](cookbooks-deduplication) cookbook for a guide). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, which means that you don't need to modify your Pulsar client code. Instead, you only need to make administrative changes (see the [Managing message deduplication](cookbooks-deduplication.md) cookbook for a guide). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.5.0/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.5.0/concepts-multi-tenancy.md index 8a17e72a4c8b3..26bd6a2fd183f 100644 --- a/site2/website/versioned_docs/version-2.5.0/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.5.0/concepts-multi-tenancy.md @@ -4,7 +4,7 @@ title: Multi Tenancy sidebar_label: "Multi Tenancy" --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -20,7 +20,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -28,7 +28,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.5.0/concepts-overview.md b/site2/website/versioned_docs/version-2.5.0/concepts-overview.md index 26730a70dd64b..bffe55dcabd60 100644 --- a/site2/website/versioned_docs/version-2.5.0/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.5.0/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.5.0/concepts-replication.md b/site2/website/versioned_docs/version-2.5.0/concepts-replication.md index d4653ca7aecd8..9cc3a7c3415bd 100644 --- a/site2/website/versioned_docs/version-2.5.0/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.5.0/concepts-replication.md @@ -8,7 +8,7 @@ Regardless of industries, when an unforeseen event occurs and brings day-to-day Pulsar's geo-replication mechanism is typically used for disaster recovery, enabling the replication of persistently stored message data across multiple data centers. For instance, your application is publishing data in one region and you would like to process it for consumption in other regions. With Pulsar’s geo-replication mechanism, messages can be produced and consumed in different geo-locations. -The diagram below illustrates the process of [geo-replication](administration-geo). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. +The diagram below illustrates the process of [geo-replication](administration-geo.md). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. ![A typical geo-replication example with full-mesh pattern](/assets/full-mesh-replication.svg) diff --git a/site2/website/versioned_docs/version-2.5.0/concepts-schema-registry.md b/site2/website/versioned_docs/version-2.5.0/concepts-schema-registry.md index d8f106a8deb8e..3593ded66167a 100644 --- a/site2/website/versioned_docs/version-2.5.0/concepts-schema-registry.md +++ b/site2/website/versioned_docs/version-2.5.0/concepts-schema-registry.md @@ -17,7 +17,7 @@ Both approaches are available in Pulsar, and you're free to adopt one or the oth #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-go.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-go.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ## Basic architecture @@ -39,7 +39,7 @@ Pulsar schemas are fairly simple data structures that consist of: ## Schema versions -In order to illustrate how schema versioning works, let's walk through an example. Imagine that the Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begin sending messages: +In order to illustrate how schema versioning works, let's walk through an example. Imagine that the Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begin sending messages: ```java diff --git a/site2/website/versioned_docs/version-2.5.0/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.5.0/concepts-tiered-storage.md index 0b45b0a2ed501..f6988e53a8cd4 100644 --- a/site2/website/versioned_docs/version-2.5.0/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.5.0/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.5.0/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.5.0/concepts-topic-compaction.md index 790ce76b7c5e0..300cf68956dd1 100644 --- a/site2/website/versioned_docs/version-2.5.0/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.5.0/concepts-topic-compaction.md @@ -6,7 +6,7 @@ sidebar_label: "Topic Compaction" Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -23,7 +23,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.5.0/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.5.0/cookbooks-compaction.md index 0720019aaa035..214e899fa086d 100644 --- a/site2/website/versioned_docs/version-2.5.0/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.5.0/cookbooks-compaction.md @@ -44,7 +44,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.5.0/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.5.0/cookbooks-non-persistent.md index d40c4fbd0cf01..1346d19a04263 100644 --- a/site2/website/versioned_docs/version-2.5.0/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.5.0/cookbooks-non-persistent.md @@ -40,7 +40,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -54,7 +54,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.5.0/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.5.0/cookbooks-partitioned.md index 25896934788fc..c857d019edc02 100644 --- a/site2/website/versioned_docs/version-2.5.0/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.5.0/cookbooks-partitioned.md @@ -3,4 +3,4 @@ id: cookbooks-partitioned title: Partitioned topics sidebar_label: "Partitioned Topics" --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.5.0/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.5.0/cookbooks-retention-expiry.md index d094c377322b0..74f30355e9e01 100644 --- a/site2/website/versioned_docs/version-2.5.0/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.5.0/cookbooks-retention-expiry.md @@ -16,7 +16,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.5.0/deploy-aws.md b/site2/website/versioned_docs/version-2.5.0/deploy-aws.md index 542f40b1187ea..0ac377375f585 100644 --- a/site2/website/versioned_docs/version-2.5.0/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.5.0/deploy-aws.md @@ -5,7 +5,7 @@ sidebar_label: "Amazon Web Services" original_id: deploy-aws --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. diff --git a/site2/website/versioned_docs/version-2.5.0/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.5.0/deploy-bare-metal-multi-cluster.md index b02ba3fbec3af..bb24a1f3d121d 100644 --- a/site2/website/versioned_docs/version-2.5.0/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.5.0/deploy-bare-metal-multi-cluster.md @@ -9,17 +9,17 @@ original_id: deploy-bare-metal-multi-cluster 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using it in a startup or on a single team, you had better opt for a single cluster. For instructions on deploying a single cluster, -see the guide [here](deploy-bare-metal). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo). Deploying a multi-cluster Pulsar instance involves the following basic steps: +A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md). Deploying a multi-cluster Pulsar instance involves the following basic steps: * Deploying two separate [ZooKeeper](#deploy-zookeeper) quorums: a [local](#deploy-local-zookeeper) quorum for each cluster in the instance and a [configuration store](#configuration-store) quorum for instance-wide tasks * Initializing [cluster metadata](#cluster-metadata-initialization) for each cluster @@ -29,7 +29,7 @@ A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You If you want to deploy a single Pulsar cluster, see [Clusters and Brokers](getting-started-standalone.md#start-the-cluster). > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). ## System requirement Currently, Pulsar is available for 64-bit **macOS**, **Linux**, and **Windows**. To use Pulsar, you need to install 64-bit JRE/JDK 8 or later versions. @@ -245,7 +245,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -364,17 +364,17 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). -You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > #### Service discovery already provided by many scheduling systems -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ### Service discovery setup -The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol) of Pulsar. +The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol.md) of Pulsar. To get started setting up the built-in service of discovery of Pulsar, you need to change a few parameters in the [`conf/discovery.conf`](reference-configuration.md#service-discovery) configuration file. Set the [`zookeeperServers`](reference-configuration.md#service-discovery-zookeeperServers) parameter to the ZooKeeper quorum connection string of the cluster and the [`configurationStoreServers`](reference-configuration.md#service-discovery-configurationStoreServers) setting to the [configuration store](reference-terminology.md#configuration-store) quorum connection string. @@ -399,7 +399,7 @@ $ bin/pulsar-daemon start discovery ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.5.0/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.5.0/deploy-bare-metal.md index 2abc2b2b5cf39..112ba4f3e634b 100644 --- a/site2/website/versioned_docs/version-2.5.0/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.5.0/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using Pulsar in a startup or on a single team, you had better opt for a single cluster. If you do need to run a multi-cluster Pulsar instance, -see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -264,9 +264,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (you had better not use a different port). -`--web-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). +`--web-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should not use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (you had better not use a different port). -`--broker-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). +`--broker-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). > If you don't have a DNS server, you can use multi-host in service URL with the following settings: @@ -391,7 +391,7 @@ webServicePortTls=8443 ### Enable Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -409,7 +409,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you want to learn more options about deploying functions worker, checkout [Deploy and manage functions worker](functions-worker). +If you want to learn more options about deploying functions worker, checkout [Deploy and manage functions worker](functions-worker.md). ### Start Brokers diff --git a/site2/website/versioned_docs/version-2.5.0/deploy-dcos.md b/site2/website/versioned_docs/version-2.5.0/deploy-dcos.md index f5f8d1faa7dbe..952d5f47e30fa 100644 --- a/site2/website/versioned_docs/version-2.5.0/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.5.0/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -If you want to enable all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of +If you want to enable all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.5.0/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.5.0/deploy-kubernetes.md index 34c2891ed43fb..95aab0b318580 100644 --- a/site2/website/versioned_docs/version-2.5.0/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.5.0/deploy-kubernetes.md @@ -7,6 +7,6 @@ original_id: deploy-kubernetes For those looking to get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -For those looking to configure and install a Pulsar cluster on Kubernetes for production usage, you should follow the complete [Installation Guide](helm-install). +For those looking to configure and install a Pulsar cluster on Kubernetes for production usage, you should follow the complete [Installation Guide](helm-install.md). diff --git a/site2/website/versioned_docs/version-2.5.0/deploy-monitoring.md b/site2/website/versioned_docs/version-2.5.0/deploy-monitoring.md index 5f926e99b7237..b3d0a3e689076 100644 --- a/site2/website/versioned_docs/version-2.5.0/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.5.0/deploy-monitoring.md @@ -73,7 +73,7 @@ The default port for bookie is `8000` (instead of `8080`). You can change the po You can use Prometheus to collect and store the metrics data. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes that needs to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is automatically setup with the [provided](deploy-kubernetes) instructions. +When you run Pulsar on bare metal, you can provide the list of nodes that needs to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is automatically setup with the [provided](deploy-kubernetes.md) instructions. ## Dashboards @@ -83,7 +83,7 @@ For that reason you only need to collect time series of metrics aggregated at th ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Dashboard](administration-dashboard). +The per-topic dashboard instructions are available at [Dashboard](administration-dashboard.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.5.0/develop-schema.md b/site2/website/versioned_docs/version-2.5.0/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.5.0/develop-schema.md +++ b/site2/website/versioned_docs/version-2.5.0/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.5.0/functions-deploy.md b/site2/website/versioned_docs/version-2.5.0/functions-deploy.md index aa4b799a0af61..a114758ee6bfe 100644 --- a/site2/website/versioned_docs/version-2.5.0/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.5.0/functions-deploy.md @@ -8,12 +8,12 @@ sidebar_label: "How-to: Deploy" To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), DC/OS, and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), DC/OS, and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface @@ -178,7 +178,7 @@ $ bin/pulsar-admin functions create \ Package management enables version management and simplifies the upgrade and rollback processes for Functions, Sinks, and Sources. When you use the same function, sink and source in different namespaces, you can upload them to a common package management system. -To use [Package management service](admin-api-packages), ensure that the package management service has been enabled in your cluster by setting the following properties in `broker.conf`. +To use [Package management service](admin-api-packages.md), ensure that the package management service has been enabled in your cluster by setting the following properties in `broker.conf`. > Note: Package management service is not enabled by default. diff --git a/site2/website/versioned_docs/version-2.5.0/functions-develop.md b/site2/website/versioned_docs/version-2.5.0/functions-develop.md index 0e8481f04e9b8..9cc77e9c26bb5 100644 --- a/site2/website/versioned_docs/version-2.5.0/functions-develop.md +++ b/site2/website/versioned_docs/version-2.5.0/functions-develop.md @@ -854,7 +854,7 @@ public class MetricRecorderFunction implements Function { ``` -> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring) guide. +> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring.md) guide.     @@ -885,11 +885,11 @@ Currently, the feature is not available in Go. ```` ### Access metrics -To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring) in Pulsar. +To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring.md) in Pulsar. ## Security -If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker). For more details, refer to [Security settings](functions-worker.md#security-settings). +If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings). Pulsar Functions can support the following providers: diff --git a/site2/website/versioned_docs/version-2.5.0/functions-overview.md b/site2/website/versioned_docs/version-2.5.0/functions-overview.md index 60cba97839af2..02f96a36a6ed2 100644 --- a/site2/website/versioned_docs/version-2.5.0/functions-overview.md +++ b/site2/website/versioned_docs/version-2.5.0/functions-overview.md @@ -164,7 +164,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.5.0/functions-worker.md b/site2/website/versioned_docs/version-2.5.0/functions-worker.md index 207eccc9f52aa..e66ba2a463c1b 100644 --- a/site2/website/versioned_docs/version-2.5.0/functions-worker.md +++ b/site2/website/versioned_docs/version-2.5.0/functions-worker.md @@ -4,7 +4,7 @@ title: Deploy and manage functions worker sidebar_label: "Setup: Pulsar Functions Worker" original_id: functions-worker --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either of the two options based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -123,7 +123,7 @@ tlsTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -150,7 +150,7 @@ properties: ``` For *Token Authentication* prodivder, add necessary settings under `properties` if needed. -See [Token Authentication](security-jwt) for more details. +See [Token Authentication](security-jwt.md) for more details. ``` diff --git a/site2/website/versioned_docs/version-2.5.0/getting-started-docker.md b/site2/website/versioned_docs/version-2.5.0/getting-started-docker.md index b9f12f0b52688..4f20971d75330 100644 --- a/site2/website/versioned_docs/version-2.5.0/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.5.0/getting-started-docker.md @@ -50,8 +50,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -110,7 +110,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.5.0/getting-started-helm.md b/site2/website/versioned_docs/version-2.5.0/getting-started-helm.md index fa521910b2e6f..7ed0a322d5cdb 100644 --- a/site2/website/versioned_docs/version-2.5.0/getting-started-helm.md +++ b/site2/website/versioned_docs/version-2.5.0/getting-started-helm.md @@ -14,7 +14,7 @@ Apache Pulsar with Helm on Kubernetes quickly, including - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, please read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, please read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -30,7 +30,7 @@ For the following steps, step 2 and step 3 are for developers and step 4 and ste ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. @@ -364,7 +364,7 @@ Then proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.5.0/getting-started-standalone.md b/site2/website/versioned_docs/version-2.5.0/getting-started-standalone.md index 3b4f8b065abd5..a5099d29359dc 100644 --- a/site2/website/versioned_docs/version-2.5.0/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.5.0/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -57,9 +57,9 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](reference-pulsar-admin). +`bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](reference-pulsar-admin.md). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.5.0/helm-install.md b/site2/website/versioned_docs/version-2.5.0/helm-install.md index 957f0dae950c8..4572238844d7c 100644 --- a/site2/website/versioned_docs/version-2.5.0/helm-install.md +++ b/site2/website/versioned_docs/version-2.5.0/helm-install.md @@ -21,7 +21,7 @@ Before proceeding to deploying Pulsar, you need to prepare your environment. ### Tools -`helm` and `kubectl` need to be [installed on your computer](helm-tools). +`helm` and `kubectl` need to be [installed on your computer](helm-tools.md). ## Cloud cluster preparation @@ -33,8 +33,8 @@ Follow the instructions to create and connect to the Kubernetes cluster of your ## Deploy Pulsar -With the environment set up and configuration generated, you can now proceed to the [deployment of Pulsar](helm-deploy). +With the environment set up and configuration generated, you can now proceed to the [deployment of Pulsar](helm-deploy.md). ## Upgrade Pulsar -If you are upgrading an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade) instead. +If you are upgrading an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade.md) instead. diff --git a/site2/website/versioned_docs/version-2.5.0/helm-overview.md b/site2/website/versioned_docs/version-2.5.0/helm-overview.md index 96e568c679969..1cb914f769dc4 100644 --- a/site2/website/versioned_docs/version-2.5.0/helm-overview.md +++ b/site2/website/versioned_docs/version-2.5.0/helm-overview.md @@ -56,12 +56,12 @@ It includes support for: For those looking to get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. This guide walks the user through deploying these charts with default values & features, but *does not* meet production ready requirements. If you wish to deploy these charts into production under sustained load, -you should follow the complete [Installation Guide](helm-install). +you should follow the complete [Installation Guide](helm-install.md). ## Troubleshooting @@ -74,7 +74,7 @@ tips and tricks for troubleshooting common issues. Please examine these first be The Apache Pulsar Helm chart contains all required dependencies. If you are just looking to deploy a Proof of Concept for testing, -we strongly suggest you follow our [Quick Start Guide](getting-started-helm) for your first iteration. +we strongly suggest you follow our [Quick Start Guide](getting-started-helm.md) for your first iteration. 1. [Preparation](helm-prepare.md) 2. [Deployment](helm-deploy.md) @@ -93,7 +93,7 @@ helm upgrade charts/pulsar -f pulsar.yaml ``` -For more detailed information, see [Upgrading](helm-upgrade). +For more detailed information, see [Upgrading](helm-upgrade.md). ## Uninstall diff --git a/site2/website/versioned_docs/version-2.5.0/helm-prepare.md b/site2/website/versioned_docs/version-2.5.0/helm-prepare.md index 83cfc17ac8aa3..bb1231f48c8ba 100644 --- a/site2/website/versioned_docs/version-2.5.0/helm-prepare.md +++ b/site2/website/versioned_docs/version-2.5.0/helm-prepare.md @@ -82,4 +82,4 @@ PROJECT= USE_LOCAL_SSD=true LOCAL_SSD_COUNT= ## Next Steps -Continue with the [installation of the chart](helm-deploy) once you have the cluster up and running. +Continue with the [installation of the chart](helm-deploy.md) once you have the cluster up and running. diff --git a/site2/website/versioned_docs/version-2.5.0/helm-tools.md b/site2/website/versioned_docs/version-2.5.0/helm-tools.md index 58d2ddcd2497a..711afa0e0f0bf 100644 --- a/site2/website/versioned_docs/version-2.5.0/helm-tools.md +++ b/site2/website/versioned_docs/version-2.5.0/helm-tools.md @@ -25,7 +25,7 @@ You can get Helm from the project's [releases page](https://github.com/helm/helm ### Next steps -Once kubectl and Helm are configured, you can continue to configuring your [Kubernetes cluster](helm-prepare). +Once kubectl and Helm are configured, you can continue to configuring your [Kubernetes cluster](helm-prepare.md). ## Additional information diff --git a/site2/website/versioned_docs/version-2.5.0/helm-upgrade.md b/site2/website/versioned_docs/version-2.5.0/helm-upgrade.md index 80778c6eb0406..8b722ab8c8a8d 100644 --- a/site2/website/versioned_docs/version-2.5.0/helm-upgrade.md +++ b/site2/website/versioned_docs/version-2.5.0/helm-upgrade.md @@ -20,7 +20,7 @@ pass this file via `-f`. Thus `helm upgrade charts/pulsar -f puls The following are the steps to upgrade Apache Pulsar to a newer version: 1. Check the change log for the specific version you would like to upgrade to -2. Go through [deployment documentation](helm-deploy) step by step +2. Go through [deployment documentation](helm-deploy.md) step by step 3. Extract your previous `--set` arguments with ```bash diff --git a/site2/website/versioned_docs/version-2.5.0/io-cdc.md b/site2/website/versioned_docs/version-2.5.0/io-cdc.md index 20f16ae0d6904..e6e662884826d 100644 --- a/site2/website/versioned_docs/version-2.5.0/io-cdc.md +++ b/site2/website/versioned_docs/version-2.5.0/io-cdc.md @@ -13,8 +13,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.5.0/io-develop.md b/site2/website/versioned_docs/version-2.5.0/io-develop.md index a957d8e6760d2..bf0f8e03d5796 100644 --- a/site2/website/versioned_docs/version-2.5.0/io-develop.md +++ b/site2/website/versioned_docs/version-2.5.0/io-develop.md @@ -8,15 +8,15 @@ original_id: io-develop This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -163,7 +163,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -192,7 +192,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). diff --git a/site2/website/versioned_docs/version-2.5.0/io-overview.md b/site2/website/versioned_docs/version-2.5.0/io-overview.md index 68df2472d6334..b538604edc46f 100644 --- a/site2/website/versioned_docs/version-2.5.0/io-overview.md +++ b/site2/website/versioned_docs/version-2.5.0/io-overview.md @@ -158,7 +158,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.5.0/io-quickstart.md b/site2/website/versioned_docs/version-2.5.0/io-quickstart.md index 1e331fa7acea6..bff40b7ac634b 100644 --- a/site2/website/versioned_docs/version-2.5.0/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.5.0/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you are able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you are able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However, all the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -109,7 +109,7 @@ This section demonstrates how to connector Pulsar to Cassandra. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink.md). ::: @@ -236,11 +236,11 @@ You can create a configuration file through one of the following methods. ``` -For more information, see [Cassandra sink connector](io-cassandra-sink). +For more information, see [Cassandra sink connector](io-cassandra-sink.md). ### Create a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on them. Run the following command to create a Cassandra sink connector with sink type _cassandra_ and the config file _examples/cassandra-sink.yml_ created previously. @@ -268,7 +268,7 @@ as a Pulsar Function and writes the messages produced in the topic _test_cassand ### Inspect a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * Get the information of a Cassandra sink. @@ -432,7 +432,7 @@ to monitor a connector and perform other operations on it. ### Delete a Cassandra Sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. ```bash @@ -451,7 +451,7 @@ This section demonstrates how to connector Pulsar to MySQL. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The JDBC sink connector pulls messages from Pulsar topics and persists the messages to MySQL or SQlite. For more information, see [JDBC sink connector](io-jdbc-sink). +* The JDBC sink connector pulls messages from Pulsar topics and persists the messages to MySQL or SQlite. For more information, see [JDBC sink connector](io-jdbc-sink.md). ::: @@ -623,7 +623,7 @@ In this section, you need to configure a JDBC sink connector. ### Create a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on it. This example creates a sink connector and specifies the desired information. @@ -669,7 +669,7 @@ The sink has been created successfully if the following message appears. ### Inspect a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * List all running JDBC sink(s). @@ -789,7 +789,7 @@ to monitor a connector and perform other operations on it. ### Stop a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to stop a connector and perform other operations on it. ```bash @@ -818,7 +818,7 @@ The sink instance has been stopped successfully if the following message disappe ### Restart a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to restart a connector and perform other operations on it. ```bash @@ -855,7 +855,7 @@ Note that `pulsar-admin sinks localrun options` **runs a sink connector locally* ### Update a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to update a connector and perform other operations on it. This example updates the parallelism of the _pulsar-mysql-jdbc-sink_ sink connector to 2. @@ -923,7 +923,7 @@ The result shows that the parallelism is 2. ### Delete a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. This example deletes the _pulsar-mysql-jdbc-sink_ sink connector. diff --git a/site2/website/versioned_docs/version-2.5.0/io-use.md b/site2/website/versioned_docs/version-2.5.0/io-use.md index 675c80c1cc647..331a5e03d9d02 100644 --- a/site2/website/versioned_docs/version-2.5.0/io-use.md +++ b/site2/website/versioned_docs/version-2.5.0/io-use.md @@ -16,7 +16,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.5.0/kubernetes-helm.md b/site2/website/versioned_docs/version-2.5.0/kubernetes-helm.md index a9f768e6a96b3..ea92a0968cd7d 100644 --- a/site2/website/versioned_docs/version-2.5.0/kubernetes-helm.md +++ b/site2/website/versioned_docs/version-2.5.0/kubernetes-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.5.0/pulsar-2.0.md b/site2/website/versioned_docs/version-2.5.0/pulsar-2.0.md index 560c8c1bd7cb7..07fba365e9ff2 100644 --- a/site2/website/versioned_docs/version-2.5.0/pulsar-2.0.md +++ b/site2/website/versioned_docs/version-2.5.0/pulsar-2.0.md @@ -4,13 +4,13 @@ title: Pulsar 2.0 sidebar_label: "Pulsar 2.0" --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.5.0/reference-cli-tools.md b/site2/website/versioned_docs/version-2.5.0/reference-cli-tools.md index 8516541f10cca..9e21a017071cd 100644 --- a/site2/website/versioned_docs/version-2.5.0/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.5.0/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.5.0/schema-manage.md b/site2/website/versioned_docs/version-2.5.0/schema-manage.md index 662b8b523726d..84d0ccdc23164 100644 --- a/site2/website/versioned_docs/version-2.5.0/schema-manage.md +++ b/site2/website/versioned_docs/version-2.5.0/schema-manage.md @@ -160,7 +160,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.5.0/schema-understand.md b/site2/website/versioned_docs/version-2.5.0/schema-understand.md index 81d6377613ca6..b245b37c940e2 100644 --- a/site2/website/versioned_docs/version-2.5.0/schema-understand.md +++ b/site2/website/versioned_docs/version-2.5.0/schema-understand.md @@ -373,7 +373,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```text diff --git a/site2/website/versioned_docs/version-2.5.0/security-athenz.md b/site2/website/versioned_docs/version-2.5.0/security-athenz.md index ba27ba43eb9d8..6d338fc4013bd 100644 --- a/site2/website/versioned_docs/version-2.5.0/security-athenz.md +++ b/site2/website/versioned_docs/version-2.5.0/security-athenz.md @@ -75,7 +75,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.5.0/security-authorization.md b/site2/website/versioned_docs/version-2.5.0/security-authorization.md index d9d33665d066b..c5e3c04b03524 100644 --- a/site2/website/versioned_docs/version-2.5.0/security-authorization.md +++ b/site2/website/versioned_docs/version-2.5.0/security-authorization.md @@ -26,7 +26,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -58,7 +58,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -86,7 +86,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication @@ -116,7 +116,7 @@ PulsarAdmin admin = PulsarAdmin.builder() When a client is identified with multiple roles in a token (the type of role claim in the token is an array) during the authentication process, Pulsar supports to check the permissions of all the roles and further authorize the client as long as one of its roles has the required permissions. > **Note**
    -> This authorization method is only compatible with [JWT authentication](security-jwt). +> This authorization method is only compatible with [JWT authentication](security-jwt.md). To enable this authorization method, configure the authorization provider as `MultiRolesTokenAuthorizationProvider` in the `conf/broker.conf` file. diff --git a/site2/website/versioned_docs/version-2.5.0/security-extending.md b/site2/website/versioned_docs/version-2.5.0/security-extending.md index 57b4c6ad928ee..37930bb286576 100644 --- a/site2/website/versioned_docs/version-2.5.0/security-extending.md +++ b/site2/website/versioned_docs/version-2.5.0/security-extending.md @@ -9,7 +9,7 @@ Pulsar provides a way to use custom authentication and authorization mechanisms. ## Authentication -Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview). +Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview.md). You can choose to use a custom authentication mechanism by providing the implementation in the form of two plugins. One plugin is for the Client library and the other plugin is for the Pulsar Broker to validate the credentials. diff --git a/site2/website/versioned_docs/version-2.5.0/security-jwt.md b/site2/website/versioned_docs/version-2.5.0/security-jwt.md index 4160d16d0e224..8ab3d1b11660c 100644 --- a/site2/website/versioned_docs/version-2.5.0/security-jwt.md +++ b/site2/website/versioned_docs/version-2.5.0/security-jwt.md @@ -31,11 +31,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -302,7 +302,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.5.0/security-kerberos.md b/site2/website/versioned_docs/version-2.5.0/security-kerberos.md index c3d6fbe044889..9d77149e845e3 100644 --- a/site2/website/versioned_docs/version-2.5.0/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.5.0/security-kerberos.md @@ -372,7 +372,7 @@ saslJaasServerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.5.0/security-overview.md b/site2/website/versioned_docs/version-2.5.0/security-overview.md index 964f579305b28..91dc68e51d9cf 100644 --- a/site2/website/versioned_docs/version-2.5.0/security-overview.md +++ b/site2/website/versioned_docs/version-2.5.0/security-overview.md @@ -17,7 +17,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.5.0/security-tls-authentication.md b/site2/website/versioned_docs/version-2.5.0/security-tls-authentication.md index 22ccf5895f39d..856362391359a 100644 --- a/site2/website/versioned_docs/version-2.5.0/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.5.0/security-tls-authentication.md @@ -7,7 +7,7 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. ### Create client certificates @@ -99,7 +99,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -119,7 +119,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.5.0/security-tls-transport.md b/site2/website/versioned_docs/version-2.5.0/security-tls-transport.md index bc10a10527374..635dc78151adf 100644 --- a/site2/website/versioned_docs/version-2.5.0/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.5.0/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,7 +27,7 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). ## Create TLS certificates @@ -128,7 +128,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Broker Configuration -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -199,7 +199,7 @@ The examples below show hostname verification being disabled for the Java client ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.5.0/security-token-admin.md b/site2/website/versioned_docs/version-2.5.0/security-token-admin.md index 01f156c6c4988..225cb8945ead4 100644 --- a/site2/website/versioned_docs/version-2.5.0/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.5.0/security-token-admin.md @@ -163,7 +163,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.5.0/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.5.0/sql-deployment-configurations.md index 0d52b0fb0aca3..1596d8e1ad938 100644 --- a/site2/website/versioned_docs/version-2.5.0/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.5.0/sql-deployment-configurations.md @@ -54,7 +54,7 @@ Since Pulsar SQL is powered by [Presto](https://prestosql.io), the configuration :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.5.0/sql-overview.md b/site2/website/versioned_docs/version-2.5.0/sql-overview.md index 2bb78260265b2..8abb23ea2acbb 100644 --- a/site2/website/versioned_docs/version-2.5.0/sql-overview.md +++ b/site2/website/versioned_docs/version-2.5.0/sql-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Overview" original_id: sql-overview --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.5.0/standalone-docker.md b/site2/website/versioned_docs/version-2.5.0/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.5.0/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.5.0/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.5.0/standalone.md b/site2/website/versioned_docs/version-2.5.0/standalone.md index fb510051b1728..c6058908889af 100644 --- a/site2/website/versioned_docs/version-2.5.0/standalone.md +++ b/site2/website/versioned_docs/version-2.5.0/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -57,9 +57,9 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](reference-pulsar-admin). +`bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](reference-pulsar-admin.md). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.5.1/admin-api-clusters.md b/site2/website/versioned_docs/version-2.5.1/admin-api-clusters.md index 04db2f60759bc..1255ad25d7aac 100644 --- a/site2/website/versioned_docs/version-2.5.1/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.5.1/admin-api-clusters.md @@ -88,7 +88,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -112,11 +112,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block > - For the latest and complete information about `Java admin API`, including classes, methods, descriptions, and more, see [Java admin API doc](https://pulsar.apache.org/api/admin/). -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -121,7 +121,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.5.1/client-libraries-cgo.md b/site2/website/versioned_docs/version-2.5.1/client-libraries-cgo.md index c79f7bb965473..f352f942b7714 100644 --- a/site2/website/versioned_docs/version-2.5.1/client-libraries-cgo.md +++ b/site2/website/versioned_docs/version-2.5.1/client-libraries-cgo.md @@ -24,7 +24,7 @@ Currently, the following Go clients are maintained in two repositories. ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Install go package @@ -57,7 +57,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -75,7 +75,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -236,14 +236,14 @@ Parameter | Description | Default `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | `Properties`| Attach a set of application defined properties to the producer. This properties will be visible in the topic stats | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | Set the number of max pending messages across all the partitions. This setting will be used to lower the max pending messages for each partition `MaxPendingMessages(int)`, if the total exceeds the configured value.| `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | `Batching` | Control whether automatic batching of messages is enabled for the producer. | false `BatchingMaxPublishDelay` | Set the time period within which the messages sent will be batched (default: 1ms) if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or until | 1ms `BatchingMaxMessages` | Set the maximum number of messages permitted in a batch. (default: 1000) If set to a value greater than 1, messages will be queued until this threshold is reached or batch interval has elapsed | 1000 @@ -512,7 +512,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.5.1/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.5.1/client-libraries-cpp.md index bb52d9effa404..684df5fd2d825 100644 --- a/site2/website/versioned_docs/version-2.5.1/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.5.1/client-libraries-cpp.md @@ -261,7 +261,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema -This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started). +This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started.md). ### Create producer with Avro schema diff --git a/site2/website/versioned_docs/version-2.5.1/client-libraries-go.md b/site2/website/versioned_docs/version-2.5.1/client-libraries-go.md index 5018346dc7e0d..64cacaa047b33 100644 --- a/site2/website/versioned_docs/version-2.5.1/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.5.1/client-libraries-go.md @@ -35,7 +35,7 @@ import "github.com/apache/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -53,7 +53,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -679,7 +679,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.5.1/client-libraries-java.md b/site2/website/versioned_docs/version-2.5.1/client-libraries-java.md index e19061882339f..d7af1f5b7c0a9 100644 --- a/site2/website/versioned_docs/version-2.5.1/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.5.1/client-libraries-java.md @@ -5,7 +5,7 @@ sidebar_label: "Java" original_id: client-libraries-java --- -You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), and [reader](#reader) of a Java client are thread-safe. @@ -14,9 +14,9 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -56,7 +56,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -82,7 +82,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -113,7 +113,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -236,7 +236,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -759,7 +759,7 @@ Total hash range size is 65536, so the max end of the range should be less than ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -873,7 +873,7 @@ The following schema formats are currently available for Java: ## Authentication -Pulsar currently supports two authentication schemes: [TLS](security-tls-authentication.md) and [Athenz](security-athenz). You can use the Pulsar Java client with both. +Pulsar currently supports two authentication schemes: [TLS](security-tls-authentication.md) and [Athenz](security-athenz.md). You can use the Pulsar Java client with both. ### TLS Authentication @@ -900,7 +900,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` diff --git a/site2/website/versioned_docs/version-2.5.1/client-libraries-node.md b/site2/website/versioned_docs/version-2.5.1/client-libraries-node.md index d2e19a824d3c6..4ef1bb53e8d13 100644 --- a/site2/website/versioned_docs/version-2.5.1/client-libraries-node.md +++ b/site2/website/versioned_docs/version-2.5.1/client-libraries-node.md @@ -44,7 +44,7 @@ Also, this library works only in Node.js 10.x or later because it uses the [`nod ::: ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here is an example for `localhost`: @@ -62,7 +62,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication), the URL will look like something like this: +If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication.md), the URL will look like something like this: ```http @@ -97,7 +97,7 @@ The following configurable parameters are available for Pulsar clients: | Parameter | Description | Default | | :-------- | :---------- | :------ | | `serviceUrl` | The connection URL for the Pulsar cluster. See [above](#connection-urls) for more info. | | -| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication) for more info. | | +| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication.md) for more info. | | | `operationTimeoutSeconds` | The timeout for Node.js client operations (creating producers, subscribing to and unsubscribing from [topics](reference-terminology.md#topic)). Retries will occur until this threshold is reached, at which point the operation will fail. | 30 | | `ioThreads` | The number of threads to use for handling connections to Pulsar [brokers](reference-terminology.md#broker). | 1 | | `messageListenerThreads` | The number of threads used by message listeners ([consumers](#consumers) and [readers](#readers)). | 1 | @@ -147,7 +147,7 @@ Pulsar Node.js producers have the following methods available: | :-------- | :---------- | :------ | | `topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer publishes messages. The topic format is `` or `//`. For example, `sample/ns1/my-topic`. | | | `producerName` | A name for the producer. If you do not explicitly assign a name, Pulsar will automatically generate a globally unique name. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | | -| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30000 | +| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30000 | | `initialSequenceId` | The initial sequence ID of the message. When producer send message, add sequence ID to message. The ID is increased each time to send. | | | `maxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `send` method will fail *unless* `blockIfQueueFull` is set to `true`. | 1000 | | `maxPendingMessagesAcrossPartitions` | The maximum size of the sum of partition's pending queue. | 50000 | diff --git a/site2/website/versioned_docs/version-2.5.1/client-libraries-python.md b/site2/website/versioned_docs/version-2.5.1/client-libraries-python.md index f9f9392ea68eb..5435a6a99a046 100644 --- a/site2/website/versioned_docs/version-2.5.1/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.5.1/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. diff --git a/site2/website/versioned_docs/version-2.5.1/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.5.1/client-libraries-websocket.md index a6e60361749de..0b31ffe78161a 100644 --- a/site2/website/versioned_docs/version-2.5.1/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.5.1/client-libraries-websocket.md @@ -4,14 +4,14 @@ title: Pulsar WebSocket API sidebar_label: "WebSocket" --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.5.1/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.5.1/concepts-architecture-overview.md index 0c7d44260ddeb..f3e75c3e307e0 100644 --- a/site2/website/versioned_docs/version-2.5.1/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.5.1/concepts-architecture-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Architecture" original_id: concepts-architecture-overview --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -17,20 +17,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers. The producers connect to the brokers to publish messages and the consumers connect to the brokers to consume the messages. -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -40,9 +40,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -136,17 +136,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -154,7 +154,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.5.1/concepts-authentication.md b/site2/website/versioned_docs/version-2.5.1/concepts-authentication.md index b375ecb16a2d8..f6307890c904a 100644 --- a/site2/website/versioned_docs/version-2.5.1/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.5.1/concepts-authentication.md @@ -5,5 +5,5 @@ sidebar_label: "Authentication and Authorization" original_id: concepts-authentication --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.5.1/concepts-clients.md b/site2/website/versioned_docs/version-2.5.1/concepts-clients.md index b68f76a2d8b08..4040624f7d636 100644 --- a/site2/website/versioned_docs/version-2.5.1/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.5.1/concepts-clients.md @@ -5,12 +5,12 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -38,7 +38,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.5.1/concepts-messaging.md b/site2/website/versioned_docs/version-2.5.1/concepts-messaging.md index ea1f55230303e..1aaac057a2484 100644 --- a/site2/website/versioned_docs/version-2.5.1/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.5.1/concepts-messaging.md @@ -15,8 +15,8 @@ Messages are the basic "unit" of Pulsar. Messages are what producers publish to Component | Purpose :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages carry raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | Messages can optionally be tagged with keys, which can be useful for things like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages carry raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | Messages can optionally be tagged with keys, which can be useful for things like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer that produced the message (producers are automatically given default names, but you can apply your own explicitly as well). Sequence ID | Each Pulsar message belongs to an ordered sequence on its topic. A message's sequence ID is its ordering in that sequence. @@ -24,7 +24,7 @@ Publish time | The timestamp of when the message was published (automatically ap Event time | An optional timestamp that applications can attach to the message representing when something happened, for example, when the message was processed. The event time of a message is 0 if none is explicitly set. TypedMessageBuilder | `TypedMessageBuilder` is used to construct a message. You can set message properties like the message key, message value with `TypedMessageBuilder`.
    When you set `TypedMessageBuilder`, the best practice is to set the key as a string. If you set the key as other types, for example, an AVRO object, the key is sent as bytes, and it is difficult to get the AVRO object back on the consumer. -> For a more in-depth breakdown of Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol). +> For a more in-depth breakdown of Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -79,7 +79,7 @@ Messages can be received from [brokers](reference-terminology.md#broker) either ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -327,7 +327,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -339,7 +339,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -377,7 +377,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -432,7 +432,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -455,12 +455,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, which means that you don't need to modify your Pulsar client code. Instead, you only need to make administrative changes (see the [Managing message deduplication](cookbooks-deduplication) cookbook for a guide). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, which means that you don't need to modify your Pulsar client code. Instead, you only need to make administrative changes (see the [Managing message deduplication](cookbooks-deduplication.md) cookbook for a guide). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.5.1/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.5.1/concepts-multi-tenancy.md index be752ccbbf9ca..c4acb21acb743 100644 --- a/site2/website/versioned_docs/version-2.5.1/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.5.1/concepts-multi-tenancy.md @@ -5,7 +5,7 @@ sidebar_label: "Multi Tenancy" original_id: concepts-multi-tenancy --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -21,7 +21,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -29,7 +29,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.5.1/concepts-overview.md b/site2/website/versioned_docs/version-2.5.1/concepts-overview.md index c1262b38daf16..e8a2f4b9d321a 100644 --- a/site2/website/versioned_docs/version-2.5.1/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.5.1/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.5.1/concepts-replication.md b/site2/website/versioned_docs/version-2.5.1/concepts-replication.md index 6e23962e99771..799f0eb4d92c6 100644 --- a/site2/website/versioned_docs/version-2.5.1/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.5.1/concepts-replication.md @@ -5,5 +5,5 @@ sidebar_label: "Geo Replication" original_id: concepts-replication --- -Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that. +Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo.md) in Pulsar enables you to do that. diff --git a/site2/website/versioned_docs/version-2.5.1/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.5.1/concepts-tiered-storage.md index 0b45b0a2ed501..f6988e53a8cd4 100644 --- a/site2/website/versioned_docs/version-2.5.1/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.5.1/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.5.1/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.5.1/concepts-topic-compaction.md index e402c965384fd..34b7ed7fbbd31 100644 --- a/site2/website/versioned_docs/version-2.5.1/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.5.1/concepts-topic-compaction.md @@ -7,7 +7,7 @@ original_id: concepts-topic-compaction Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -24,7 +24,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.5.1/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.5.1/cookbooks-compaction.md index f436568e393de..864fb75255bfa 100644 --- a/site2/website/versioned_docs/version-2.5.1/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.5.1/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.5.1/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.5.1/cookbooks-non-persistent.md index d40c4fbd0cf01..1346d19a04263 100644 --- a/site2/website/versioned_docs/version-2.5.1/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.5.1/cookbooks-non-persistent.md @@ -40,7 +40,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -54,7 +54,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.5.1/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.5.1/cookbooks-partitioned.md index 4649d84057c84..cdf3bf3d14005 100644 --- a/site2/website/versioned_docs/version-2.5.1/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.5.1/cookbooks-partitioned.md @@ -7,7 +7,7 @@ original_id: cookbooks-partitioned By default, Pulsar topics are served by a single broker. Using only a single broker limits a topic's maximum throughput. *Partitioned topics* are a special type of topic that can span multiple brokers and thus allow for much higher throughput. For an explanation of how partitioned topics work, see the [Partitioned Topics](concepts-messaging.md#partitioned-topics) concepts. -You can publish to partitioned topics using Pulsar client libraries and you can [create and manage](#managing-partitioned-topics) partitioned topics using Pulsar [admin API](admin-api-overview). +You can publish to partitioned topics using Pulsar client libraries and you can [create and manage](#managing-partitioned-topics) partitioned topics using Pulsar [admin API](admin-api-overview.md). ## Publish to partitioned topics @@ -100,4 +100,4 @@ If a message has a key, it supersedes the round robin routing policy. The follow ## Manage partitioned topics -You can use Pulsar [admin API](admin-api-overview.md) to create and manage [partitioned topics](admin-api-partitioned-topics). \ No newline at end of file +You can use Pulsar [admin API](admin-api-overview.md) to create and manage [partitioned topics](admin-api-partitioned-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.5.1/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.5.1/cookbooks-retention-expiry.md index 86160c3f7ecf8..cd764fad6eb37 100644 --- a/site2/website/versioned_docs/version-2.5.1/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.5.1/cookbooks-retention-expiry.md @@ -21,7 +21,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.5.1/deploy-aws.md b/site2/website/versioned_docs/version-2.5.1/deploy-aws.md index 5db846fe790f2..60ac4c22d7672 100644 --- a/site2/website/versioned_docs/version-2.5.1/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.5.1/deploy-aws.md @@ -4,7 +4,7 @@ title: Deploying a Pulsar cluster on AWS using Terraform and Ansible sidebar_label: "Amazon Web Services" --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -209,7 +209,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.5.1/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.5.1/deploy-bare-metal-multi-cluster.md index 70dff45080b03..8a765d8843f43 100644 --- a/site2/website/versioned_docs/version-2.5.1/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.5.1/deploy-bare-metal-multi-cluster.md @@ -6,13 +6,13 @@ sidebar_label: "Bare metal multi-cluster" :::tip -1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster). -2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster.md). +2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo).Deploying a multi-cluster Pulsar instance consists of the following steps: +A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md).Deploying a multi-cluster Pulsar instance consists of the following steps: 1. Deploying two separate ZooKeeper quorums: a local quorum for each cluster in the instance and a configuration store quorum for instance-wide tasks 2. Initializing cluster metadata for each cluster @@ -21,7 +21,7 @@ A Pulsar instance consists of multiple Pulsar clusters working in unison. You ca > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. ## System requirement @@ -65,7 +65,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -244,7 +244,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -363,16 +363,16 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. -You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > **Service discovery already provided by many scheduling systems** -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.5.1/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.5.1/deploy-bare-metal.md index 4565eada114b7..fde30926519b1 100644 --- a/site2/website/versioned_docs/version-2.5.1/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.5.1/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using Pulsar in a startup or on a single team, it is simplest to opt for a single cluster. If you do need to run a multi-cluster Pulsar instance, -see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -263,9 +263,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (you had better not use a different port). -`--web-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). +`--web-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should not use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (you had better not use a different port). -`--broker-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). +`--broker-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). > If you do not have a DNS server, you can use multi-host format in the service URL with the following settings: @@ -389,7 +389,7 @@ webServicePortTls=8443 ### Enable Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -407,7 +407,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker). +If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker.md). ### Start Brokers diff --git a/site2/website/versioned_docs/version-2.5.1/deploy-dcos.md b/site2/website/versioned_docs/version-2.5.1/deploy-dcos.md index 07f446edaffd6..599d3a1b0454c 100644 --- a/site2/website/versioned_docs/version-2.5.1/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.5.1/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -To enable all built-in [Pulsar IO](io-overview) connectors in your Pulsar deploymente, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). +To enable all built-in [Pulsar IO](io-overview.md) connectors in your Pulsar deploymente, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.5.1/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.5.1/deploy-kubernetes.md index 4e170dcd4711d..7455247c901d4 100644 --- a/site2/website/versioned_docs/version-2.5.1/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.5.1/deploy-kubernetes.md @@ -5,6 +5,6 @@ sidebar_label: "Kubernetes" --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.5.1/deploy-monitoring.md b/site2/website/versioned_docs/version-2.5.1/deploy-monitoring.md index 2efc328097c00..40e918d9434fd 100644 --- a/site2/website/versioned_docs/version-2.5.1/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.5.1/deploy-monitoring.md @@ -73,7 +73,7 @@ The default port for bookie is `8000` (instead of `8080`). You can change the po You can use Prometheus to collect and store the metrics data. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes that needs to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is automatically setup with the [provided](deploy-kubernetes) instructions. +When you run Pulsar on bare metal, you can provide the list of nodes that needs to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is automatically setup with the [provided](deploy-kubernetes.md) instructions. ## Dashboards @@ -83,7 +83,7 @@ For that reason you only need to collect time series of metrics aggregated at th ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Dashboard](administration-dashboard). +The per-topic dashboard instructions are available at [Dashboard](administration-dashboard.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.5.1/develop-schema.md b/site2/website/versioned_docs/version-2.5.1/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.5.1/develop-schema.md +++ b/site2/website/versioned_docs/version-2.5.1/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.5.1/functions-deploy.md b/site2/website/versioned_docs/version-2.5.1/functions-deploy.md index aa4b799a0af61..a114758ee6bfe 100644 --- a/site2/website/versioned_docs/version-2.5.1/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.5.1/functions-deploy.md @@ -8,12 +8,12 @@ sidebar_label: "How-to: Deploy" To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), DC/OS, and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), DC/OS, and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface @@ -178,7 +178,7 @@ $ bin/pulsar-admin functions create \ Package management enables version management and simplifies the upgrade and rollback processes for Functions, Sinks, and Sources. When you use the same function, sink and source in different namespaces, you can upload them to a common package management system. -To use [Package management service](admin-api-packages), ensure that the package management service has been enabled in your cluster by setting the following properties in `broker.conf`. +To use [Package management service](admin-api-packages.md), ensure that the package management service has been enabled in your cluster by setting the following properties in `broker.conf`. > Note: Package management service is not enabled by default. diff --git a/site2/website/versioned_docs/version-2.5.1/functions-develop.md b/site2/website/versioned_docs/version-2.5.1/functions-develop.md index fc869e9362b7d..c61ee279d6cb7 100644 --- a/site2/website/versioned_docs/version-2.5.1/functions-develop.md +++ b/site2/website/versioned_docs/version-2.5.1/functions-develop.md @@ -854,7 +854,7 @@ public class MetricRecorderFunction implements Function { ``` -> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring) guide. +> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring.md) guide.     @@ -885,11 +885,11 @@ Currently, the feature is not available in Go. ```` ### Access metrics -To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring) in Pulsar. +To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring.md) in Pulsar. ## Security -If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker). For more details, refer to [Security settings](functions-worker.md#security-settings). +If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings). Pulsar Functions can support the following providers: diff --git a/site2/website/versioned_docs/version-2.5.1/functions-overview.md b/site2/website/versioned_docs/version-2.5.1/functions-overview.md index 25dc6029acf70..816d301e0fd0e 100644 --- a/site2/website/versioned_docs/version-2.5.1/functions-overview.md +++ b/site2/website/versioned_docs/version-2.5.1/functions-overview.md @@ -164,7 +164,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.5.1/functions-worker.md b/site2/website/versioned_docs/version-2.5.1/functions-worker.md index cfeab2b980c35..72ed20cc62c5d 100644 --- a/site2/website/versioned_docs/version-2.5.1/functions-worker.md +++ b/site2/website/versioned_docs/version-2.5.1/functions-worker.md @@ -3,7 +3,7 @@ id: functions-worker title: Deploy and manage functions worker sidebar_label: "Setup: Pulsar Functions Worker" --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -198,7 +198,7 @@ brokerClientTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -218,7 +218,7 @@ authenticationProviders: [ provider1, provider2 ] ``` For *TLS Authentication* provider, follow the example below to add the necessary settings. -See [TLS Authentication](security-tls-authentication) for more details. +See [TLS Authentication](security-tls-authentication.md) for more details. ``` @@ -242,7 +242,7 @@ properties: ``` For *Token Authentication* provider, add necessary settings for `properties` if needed. -See [Token Authentication](security-jwt) for more details. +See [Token Authentication](security-jwt.md) for more details. Note: key files must be DER-encoded ``` @@ -281,7 +281,7 @@ superUserRoles: You can use the public and private key pair that the application configures to perform encryption. Only the consumers with a valid key can decrypt the encrypted messages. -To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption). +To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption.md). We include the relevant configuration information of `CryptoConfig` into `ProducerConfig`. The specific configurable field information about `CryptoConfig` is as follows: diff --git a/site2/website/versioned_docs/version-2.5.1/getting-started-docker.md b/site2/website/versioned_docs/version-2.5.1/getting-started-docker.md index b9f12f0b52688..4f20971d75330 100644 --- a/site2/website/versioned_docs/version-2.5.1/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.5.1/getting-started-docker.md @@ -50,8 +50,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -110,7 +110,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.5.1/getting-started-helm.md b/site2/website/versioned_docs/version-2.5.1/getting-started-helm.md index 65a9fc04fe568..440087c275c05 100644 --- a/site2/website/versioned_docs/version-2.5.1/getting-started-helm.md +++ b/site2/website/versioned_docs/version-2.5.1/getting-started-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.5.1/getting-started-standalone.md b/site2/website/versioned_docs/version-2.5.1/getting-started-standalone.md index 95782fc6ca8e1..05babc33a8bdb 100644 --- a/site2/website/versioned_docs/version-2.5.1/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.5.1/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -65,7 +65,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -74,7 +74,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -123,7 +123,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -137,7 +137,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -170,12 +170,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -208,7 +208,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.5.1/helm-install.md b/site2/website/versioned_docs/version-2.5.1/helm-install.md index dd9c8afdad185..16a646d9aa56f 100644 --- a/site2/website/versioned_docs/version-2.5.1/helm-install.md +++ b/site2/website/versioned_docs/version-2.5.1/helm-install.md @@ -20,7 +20,7 @@ Before deploying Pulsar, you need to prepare your environment. ### Tools -Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools) on your computer. +Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools.md) on your computer. ## Cloud cluster preparation @@ -30,8 +30,8 @@ To create and connect to the Kubernetes cluster, follow the instructions: ## Pulsar deployment -Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy). +Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy.md). ## Pulsar upgrade -To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade). +To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade.md). diff --git a/site2/website/versioned_docs/version-2.5.1/helm-overview.md b/site2/website/versioned_docs/version-2.5.1/helm-overview.md index 9afa1533bbab4..9aa05859c0449 100644 --- a/site2/website/versioned_docs/version-2.5.1/helm-overview.md +++ b/site2/website/versioned_docs/version-2.5.1/helm-overview.md @@ -51,9 +51,9 @@ Moreover, Helm chart supports: ## Quick start -To run with Apache Pulsar Helm chart as fast as possible in a **non-production** use case, we provide a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +To run with Apache Pulsar Helm chart as fast as possible in a **non-production** use case, we provide a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -This guide walks you through deploying Apache Pulsar Helm chart with default values and features, but it is *not* suitable for deployments in production-ready environments. To deploy the charts in production under sustained load, you can follow the complete [Installation Guide](helm-install). +This guide walks you through deploying Apache Pulsar Helm chart with default values and features, but it is *not* suitable for deployments in production-ready environments. To deploy the charts in production under sustained load, you can follow the complete [Installation Guide](helm-install.md). ## Troubleshooting @@ -63,7 +63,7 @@ Although we have done our best to make these charts as seamless as possible, tro The Apache Pulsar Helm chart contains all required dependencies. -If you deploy a PoC for testing, we strongly suggest you follow this [Quick Start Guide](getting-started-helm) for your first iteration. +If you deploy a PoC for testing, we strongly suggest you follow this [Quick Start Guide](getting-started-helm.md) for your first iteration. 1. [Preparation](helm-prepare.md) 2. [Deployment](helm-deploy.md) @@ -81,7 +81,7 @@ helm upgrade apache/pulsar -f pulsar.yaml ``` -For more detailed information, see [Upgrading](helm-upgrade). +For more detailed information, see [Upgrading](helm-upgrade.md). ## Uninstallation diff --git a/site2/website/versioned_docs/version-2.5.1/helm-tools.md b/site2/website/versioned_docs/version-2.5.1/helm-tools.md index 7a9527160a131..c594990176407 100644 --- a/site2/website/versioned_docs/version-2.5.1/helm-tools.md +++ b/site2/website/versioned_docs/version-2.5.1/helm-tools.md @@ -24,7 +24,7 @@ You can get Helm from the project's [releases page](https://github.com/helm/helm ### Next steps -Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare). +Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare.md). ## Additional information diff --git a/site2/website/versioned_docs/version-2.5.1/helm-upgrade.md b/site2/website/versioned_docs/version-2.5.1/helm-upgrade.md index 9713557f8ba24..71b2ac4a7f6d4 100644 --- a/site2/website/versioned_docs/version-2.5.1/helm-upgrade.md +++ b/site2/website/versioned_docs/version-2.5.1/helm-upgrade.md @@ -19,7 +19,7 @@ You can retrieve your previous `--set` arguments cleanly, with `helm get values To upgrade Apache Pulsar to a newer version, follow these steps: 1. Check the change log for the specific version you would like to upgrade to. -2. Go through [deployment documentation](helm-deploy) step by step. +2. Go through [deployment documentation](helm-deploy.md) step by step. 3. Extract your previous `--set` arguments with the following command. ```bash diff --git a/site2/website/versioned_docs/version-2.5.1/io-cdc.md b/site2/website/versioned_docs/version-2.5.1/io-cdc.md index df373977e3eaa..a3527dc0a9d65 100644 --- a/site2/website/versioned_docs/version-2.5.1/io-cdc.md +++ b/site2/website/versioned_docs/version-2.5.1/io-cdc.md @@ -12,8 +12,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.5.1/io-develop.md b/site2/website/versioned_docs/version-2.5.1/io-develop.md index 4a815b3b9d6f6..7a356662560c9 100644 --- a/site2/website/versioned_docs/version-2.5.1/io-develop.md +++ b/site2/website/versioned_docs/version-2.5.1/io-develop.md @@ -8,15 +8,15 @@ original_id: io-develop This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -163,7 +163,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -192,7 +192,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). diff --git a/site2/website/versioned_docs/version-2.5.1/io-overview.md b/site2/website/versioned_docs/version-2.5.1/io-overview.md index 2a6ecaee5883f..ea49e4b042e8c 100644 --- a/site2/website/versioned_docs/version-2.5.1/io-overview.md +++ b/site2/website/versioned_docs/version-2.5.1/io-overview.md @@ -157,7 +157,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.5.1/io-quickstart.md b/site2/website/versioned_docs/version-2.5.1/io-quickstart.md index 0c9c78caec8a0..6ebf49d0f88b7 100644 --- a/site2/website/versioned_docs/version-2.5.1/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.5.1/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you are able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you are able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However, all the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -109,7 +109,7 @@ This section demonstrates how to connector Pulsar to Cassandra. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink.md). ::: @@ -236,11 +236,11 @@ You can create a configuration file through one of the following methods. ``` -For more information, see [Cassandra sink connector](io-cassandra-sink). +For more information, see [Cassandra sink connector](io-cassandra-sink.md). ### Create a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on them. Run the following command to create a Cassandra sink connector with sink type _cassandra_ and the config file _examples/cassandra-sink.yml_ created previously. @@ -268,7 +268,7 @@ as a Pulsar Function and writes the messages produced in the topic _test_cassand ### Inspect a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * Get the information of a Cassandra sink. @@ -432,7 +432,7 @@ to monitor a connector and perform other operations on it. ### Delete a Cassandra Sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. ```bash @@ -451,7 +451,7 @@ This section demonstrates how to connector Pulsar to MySQL. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The JDBC sink connector pulls messages from Pulsar topics and persists the messages to MySQL or SQlite. For more information, see [JDBC sink connector](io-jdbc-sink). +* The JDBC sink connector pulls messages from Pulsar topics and persists the messages to MySQL or SQlite. For more information, see [JDBC sink connector](io-jdbc-sink.md). ::: @@ -624,7 +624,7 @@ In this section, you need to configure a JDBC sink connector. ### Create a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on it. This example creates a sink connector and specifies the desired information. @@ -670,7 +670,7 @@ The sink has been created successfully if the following message appears. ### Inspect a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * List all running JDBC sink(s). @@ -790,7 +790,7 @@ to monitor a connector and perform other operations on it. ### Stop a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to stop a connector and perform other operations on it. ```bash @@ -819,7 +819,7 @@ The sink instance has been stopped successfully if the following message disappe ### Restart a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to restart a connector and perform other operations on it. ```bash @@ -856,7 +856,7 @@ Note that `pulsar-admin sinks localrun options` **runs a sink connector locally* ### Update a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to update a connector and perform other operations on it. This example updates the parallelism of the _pulsar-mysql-jdbc-sink_ sink connector to 2. @@ -924,7 +924,7 @@ The result shows that the parallelism is 2. ### Delete a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. This example deletes the _pulsar-mysql-jdbc-sink_ sink connector. diff --git a/site2/website/versioned_docs/version-2.5.1/io-use.md b/site2/website/versioned_docs/version-2.5.1/io-use.md index d8c5aaa5c8d88..6760a5f61a125 100644 --- a/site2/website/versioned_docs/version-2.5.1/io-use.md +++ b/site2/website/versioned_docs/version-2.5.1/io-use.md @@ -14,7 +14,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.5.1/pulsar-2.0.md b/site2/website/versioned_docs/version-2.5.1/pulsar-2.0.md index 11c5e66cb62c2..752590f57b558 100644 --- a/site2/website/versioned_docs/version-2.5.1/pulsar-2.0.md +++ b/site2/website/versioned_docs/version-2.5.1/pulsar-2.0.md @@ -5,13 +5,13 @@ sidebar_label: "Pulsar 2.0" original_id: pulsar-2.0 --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.5.1/reference-cli-tools.md b/site2/website/versioned_docs/version-2.5.1/reference-cli-tools.md index 8516541f10cca..9e21a017071cd 100644 --- a/site2/website/versioned_docs/version-2.5.1/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.5.1/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.5.1/schema-get-started.md b/site2/website/versioned_docs/version-2.5.1/schema-get-started.md index 97e270586a26d..afacb0fa51f2e 100644 --- a/site2/website/versioned_docs/version-2.5.1/schema-get-started.md +++ b/site2/website/versioned_docs/version-2.5.1/schema-get-started.md @@ -17,7 +17,7 @@ Applications typically adopt one of the following approaches to guarantee type s #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ### Client-side approach diff --git a/site2/website/versioned_docs/version-2.5.1/schema-manage.md b/site2/website/versioned_docs/version-2.5.1/schema-manage.md index a280eba263e27..580598c6edd38 100644 --- a/site2/website/versioned_docs/version-2.5.1/schema-manage.md +++ b/site2/website/versioned_docs/version-2.5.1/schema-manage.md @@ -160,7 +160,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.5.1/schema-understand.md b/site2/website/versioned_docs/version-2.5.1/schema-understand.md index fda3d6dfecb4c..2b5fd1f44b360 100644 --- a/site2/website/versioned_docs/version-2.5.1/schema-understand.md +++ b/site2/website/versioned_docs/version-2.5.1/schema-understand.md @@ -372,7 +372,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```java diff --git a/site2/website/versioned_docs/version-2.5.1/security-athenz.md b/site2/website/versioned_docs/version-2.5.1/security-athenz.md index ba27ba43eb9d8..6d338fc4013bd 100644 --- a/site2/website/versioned_docs/version-2.5.1/security-athenz.md +++ b/site2/website/versioned_docs/version-2.5.1/security-athenz.md @@ -75,7 +75,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.5.1/security-authorization.md b/site2/website/versioned_docs/version-2.5.1/security-authorization.md index d9d33665d066b..c5e3c04b03524 100644 --- a/site2/website/versioned_docs/version-2.5.1/security-authorization.md +++ b/site2/website/versioned_docs/version-2.5.1/security-authorization.md @@ -26,7 +26,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -58,7 +58,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -86,7 +86,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication @@ -116,7 +116,7 @@ PulsarAdmin admin = PulsarAdmin.builder() When a client is identified with multiple roles in a token (the type of role claim in the token is an array) during the authentication process, Pulsar supports to check the permissions of all the roles and further authorize the client as long as one of its roles has the required permissions. > **Note**
    -> This authorization method is only compatible with [JWT authentication](security-jwt). +> This authorization method is only compatible with [JWT authentication](security-jwt.md). To enable this authorization method, configure the authorization provider as `MultiRolesTokenAuthorizationProvider` in the `conf/broker.conf` file. diff --git a/site2/website/versioned_docs/version-2.5.1/security-bouncy-castle.md b/site2/website/versioned_docs/version-2.5.1/security-bouncy-castle.md index 1012f0e4933b6..eaf4f907f17e4 100644 --- a/site2/website/versioned_docs/version-2.5.1/security-bouncy-castle.md +++ b/site2/website/versioned_docs/version-2.5.1/security-bouncy-castle.md @@ -16,7 +16,7 @@ In Pulsar, security and crypto have dependencies on BouncyCastle Jars. For the d `Bouncy Castle` provides both [FIPS](https://www.bouncycastle.org/fips_faq.html) and non-FIPS version. But in a JVM, you can not include both of the 2 versions, and you need to exclude the current version before include the other. -In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. +In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption.md). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. ## Include dependencies of BC-non-FIPS diff --git a/site2/website/versioned_docs/version-2.5.1/security-jwt.md b/site2/website/versioned_docs/version-2.5.1/security-jwt.md index 4157a6246d1fc..e3a52e6f058e1 100644 --- a/site2/website/versioned_docs/version-2.5.1/security-jwt.md +++ b/site2/website/versioned_docs/version-2.5.1/security-jwt.md @@ -32,11 +32,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -288,7 +288,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.5.1/security-kerberos.md b/site2/website/versioned_docs/version-2.5.1/security-kerberos.md index c3d6fbe044889..9d77149e845e3 100644 --- a/site2/website/versioned_docs/version-2.5.1/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.5.1/security-kerberos.md @@ -372,7 +372,7 @@ saslJaasServerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.5.1/security-overview.md b/site2/website/versioned_docs/version-2.5.1/security-overview.md index 2f3fc035ad4cb..3f08cb29ad854 100644 --- a/site2/website/versioned_docs/version-2.5.1/security-overview.md +++ b/site2/website/versioned_docs/version-2.5.1/security-overview.md @@ -20,7 +20,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.5.1/security-tls-authentication.md b/site2/website/versioned_docs/version-2.5.1/security-tls-authentication.md index 2987325ec3221..3a3e35b731000 100644 --- a/site2/website/versioned_docs/version-2.5.1/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.5.1/security-tls-authentication.md @@ -7,9 +7,9 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -101,7 +101,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -121,7 +121,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.5.1/security-tls-transport.md b/site2/website/versioned_docs/version-2.5.1/security-tls-transport.md index a1a9f3955daaf..9fbbe2bae2cab 100644 --- a/site2/website/versioned_docs/version-2.5.1/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.5.1/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,9 +27,9 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). -`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ## Create TLS certificates @@ -130,7 +130,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Broker Configuration -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -201,7 +201,7 @@ The examples below show hostname verification being disabled for the Java client ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.5.1/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.5.1/sql-deployment-configurations.md index 639ac988b3409..069c6ac9c92da 100644 --- a/site2/website/versioned_docs/version-2.5.1/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.5.1/sql-deployment-configurations.md @@ -141,7 +141,7 @@ Since Pulsar SQL is powered by [Trino (formerly Presto SQL)](https://trino.io), :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.5.1/sql-overview.md b/site2/website/versioned_docs/version-2.5.1/sql-overview.md index 2f827f4267933..5b01dfde56e7f 100644 --- a/site2/website/versioned_docs/version-2.5.1/sql-overview.md +++ b/site2/website/versioned_docs/version-2.5.1/sql-overview.md @@ -4,7 +4,7 @@ title: Pulsar SQL Overview sidebar_label: "Overview" --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL)](https://trino.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL.md)](https://trino.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.5.1/standalone.md b/site2/website/versioned_docs/version-2.5.1/standalone.md index 2ff2d6059a284..292b344effe6e 100644 --- a/site2/website/versioned_docs/version-2.5.1/standalone.md +++ b/site2/website/versioned_docs/version-2.5.1/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -65,7 +65,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -74,7 +74,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -123,7 +123,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -137,7 +137,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -170,12 +170,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -208,7 +208,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.5.2/admin-api-clusters.md b/site2/website/versioned_docs/version-2.5.2/admin-api-clusters.md index 04db2f60759bc..1255ad25d7aac 100644 --- a/site2/website/versioned_docs/version-2.5.2/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.5.2/admin-api-clusters.md @@ -88,7 +88,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -112,11 +112,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block > - For the latest and complete information about `Java admin API`, including classes, methods, descriptions, and more, see [Java admin API doc](https://pulsar.apache.org/api/admin/). -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -121,7 +121,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.5.2/client-libraries-cgo.md b/site2/website/versioned_docs/version-2.5.2/client-libraries-cgo.md index c79f7bb965473..f352f942b7714 100644 --- a/site2/website/versioned_docs/version-2.5.2/client-libraries-cgo.md +++ b/site2/website/versioned_docs/version-2.5.2/client-libraries-cgo.md @@ -24,7 +24,7 @@ Currently, the following Go clients are maintained in two repositories. ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Install go package @@ -57,7 +57,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -75,7 +75,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -236,14 +236,14 @@ Parameter | Description | Default `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | `Properties`| Attach a set of application defined properties to the producer. This properties will be visible in the topic stats | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | Set the number of max pending messages across all the partitions. This setting will be used to lower the max pending messages for each partition `MaxPendingMessages(int)`, if the total exceeds the configured value.| `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | `Batching` | Control whether automatic batching of messages is enabled for the producer. | false `BatchingMaxPublishDelay` | Set the time period within which the messages sent will be batched (default: 1ms) if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or until | 1ms `BatchingMaxMessages` | Set the maximum number of messages permitted in a batch. (default: 1000) If set to a value greater than 1, messages will be queued until this threshold is reached or batch interval has elapsed | 1000 @@ -512,7 +512,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.5.2/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.5.2/client-libraries-cpp.md index bb52d9effa404..684df5fd2d825 100644 --- a/site2/website/versioned_docs/version-2.5.2/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.5.2/client-libraries-cpp.md @@ -261,7 +261,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema -This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started). +This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started.md). ### Create producer with Avro schema diff --git a/site2/website/versioned_docs/version-2.5.2/client-libraries-go.md b/site2/website/versioned_docs/version-2.5.2/client-libraries-go.md index 5018346dc7e0d..64cacaa047b33 100644 --- a/site2/website/versioned_docs/version-2.5.2/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.5.2/client-libraries-go.md @@ -35,7 +35,7 @@ import "github.com/apache/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -53,7 +53,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -679,7 +679,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.5.2/client-libraries-java.md b/site2/website/versioned_docs/version-2.5.2/client-libraries-java.md index e19061882339f..d7af1f5b7c0a9 100644 --- a/site2/website/versioned_docs/version-2.5.2/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.5.2/client-libraries-java.md @@ -5,7 +5,7 @@ sidebar_label: "Java" original_id: client-libraries-java --- -You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), and [reader](#reader) of a Java client are thread-safe. @@ -14,9 +14,9 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -56,7 +56,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -82,7 +82,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -113,7 +113,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -236,7 +236,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -759,7 +759,7 @@ Total hash range size is 65536, so the max end of the range should be less than ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -873,7 +873,7 @@ The following schema formats are currently available for Java: ## Authentication -Pulsar currently supports two authentication schemes: [TLS](security-tls-authentication.md) and [Athenz](security-athenz). You can use the Pulsar Java client with both. +Pulsar currently supports two authentication schemes: [TLS](security-tls-authentication.md) and [Athenz](security-athenz.md). You can use the Pulsar Java client with both. ### TLS Authentication @@ -900,7 +900,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` diff --git a/site2/website/versioned_docs/version-2.5.2/client-libraries-node.md b/site2/website/versioned_docs/version-2.5.2/client-libraries-node.md index d2e19a824d3c6..4ef1bb53e8d13 100644 --- a/site2/website/versioned_docs/version-2.5.2/client-libraries-node.md +++ b/site2/website/versioned_docs/version-2.5.2/client-libraries-node.md @@ -44,7 +44,7 @@ Also, this library works only in Node.js 10.x or later because it uses the [`nod ::: ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here is an example for `localhost`: @@ -62,7 +62,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication), the URL will look like something like this: +If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication.md), the URL will look like something like this: ```http @@ -97,7 +97,7 @@ The following configurable parameters are available for Pulsar clients: | Parameter | Description | Default | | :-------- | :---------- | :------ | | `serviceUrl` | The connection URL for the Pulsar cluster. See [above](#connection-urls) for more info. | | -| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication) for more info. | | +| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication.md) for more info. | | | `operationTimeoutSeconds` | The timeout for Node.js client operations (creating producers, subscribing to and unsubscribing from [topics](reference-terminology.md#topic)). Retries will occur until this threshold is reached, at which point the operation will fail. | 30 | | `ioThreads` | The number of threads to use for handling connections to Pulsar [brokers](reference-terminology.md#broker). | 1 | | `messageListenerThreads` | The number of threads used by message listeners ([consumers](#consumers) and [readers](#readers)). | 1 | @@ -147,7 +147,7 @@ Pulsar Node.js producers have the following methods available: | :-------- | :---------- | :------ | | `topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer publishes messages. The topic format is `` or `//`. For example, `sample/ns1/my-topic`. | | | `producerName` | A name for the producer. If you do not explicitly assign a name, Pulsar will automatically generate a globally unique name. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | | -| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30000 | +| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30000 | | `initialSequenceId` | The initial sequence ID of the message. When producer send message, add sequence ID to message. The ID is increased each time to send. | | | `maxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `send` method will fail *unless* `blockIfQueueFull` is set to `true`. | 1000 | | `maxPendingMessagesAcrossPartitions` | The maximum size of the sum of partition's pending queue. | 50000 | diff --git a/site2/website/versioned_docs/version-2.5.2/client-libraries-python.md b/site2/website/versioned_docs/version-2.5.2/client-libraries-python.md index f9f9392ea68eb..5435a6a99a046 100644 --- a/site2/website/versioned_docs/version-2.5.2/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.5.2/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. diff --git a/site2/website/versioned_docs/version-2.5.2/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.5.2/client-libraries-websocket.md index a6e60361749de..0b31ffe78161a 100644 --- a/site2/website/versioned_docs/version-2.5.2/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.5.2/client-libraries-websocket.md @@ -4,14 +4,14 @@ title: Pulsar WebSocket API sidebar_label: "WebSocket" --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.5.2/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.5.2/concepts-architecture-overview.md index 0d96b59f8ae91..1ce346fc16941 100644 --- a/site2/website/versioned_docs/version-2.5.2/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.5.2/concepts-architecture-overview.md @@ -4,7 +4,7 @@ title: Architecture Overview sidebar_label: "Architecture" --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -16,20 +16,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers. The producers connect to the brokers to publish messages and the consumers connect to the brokers to consume the messages. -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -39,9 +39,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -139,17 +139,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -157,7 +157,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.5.2/concepts-authentication.md b/site2/website/versioned_docs/version-2.5.2/concepts-authentication.md index 335da8dc5a519..fbefead69f382 100644 --- a/site2/website/versioned_docs/version-2.5.2/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.5.2/concepts-authentication.md @@ -4,5 +4,5 @@ title: Authentication and Authorization sidebar_label: "Authentication and Authorization" --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.5.2/concepts-clients.md b/site2/website/versioned_docs/version-2.5.2/concepts-clients.md index 65201b5ce1cd2..c0fe61cd06e76 100644 --- a/site2/website/versioned_docs/version-2.5.2/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.5.2/concepts-clients.md @@ -4,12 +4,12 @@ title: Pulsar Clients sidebar_label: "Clients" --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -37,7 +37,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.5.2/concepts-messaging.md b/site2/website/versioned_docs/version-2.5.2/concepts-messaging.md index ea1f55230303e..1aaac057a2484 100644 --- a/site2/website/versioned_docs/version-2.5.2/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.5.2/concepts-messaging.md @@ -15,8 +15,8 @@ Messages are the basic "unit" of Pulsar. Messages are what producers publish to Component | Purpose :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages carry raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | Messages can optionally be tagged with keys, which can be useful for things like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages carry raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | Messages can optionally be tagged with keys, which can be useful for things like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer that produced the message (producers are automatically given default names, but you can apply your own explicitly as well). Sequence ID | Each Pulsar message belongs to an ordered sequence on its topic. A message's sequence ID is its ordering in that sequence. @@ -24,7 +24,7 @@ Publish time | The timestamp of when the message was published (automatically ap Event time | An optional timestamp that applications can attach to the message representing when something happened, for example, when the message was processed. The event time of a message is 0 if none is explicitly set. TypedMessageBuilder | `TypedMessageBuilder` is used to construct a message. You can set message properties like the message key, message value with `TypedMessageBuilder`.
    When you set `TypedMessageBuilder`, the best practice is to set the key as a string. If you set the key as other types, for example, an AVRO object, the key is sent as bytes, and it is difficult to get the AVRO object back on the consumer. -> For a more in-depth breakdown of Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol). +> For a more in-depth breakdown of Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -79,7 +79,7 @@ Messages can be received from [brokers](reference-terminology.md#broker) either ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -327,7 +327,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -339,7 +339,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -377,7 +377,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -432,7 +432,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -455,12 +455,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, which means that you don't need to modify your Pulsar client code. Instead, you only need to make administrative changes (see the [Managing message deduplication](cookbooks-deduplication) cookbook for a guide). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, which means that you don't need to modify your Pulsar client code. Instead, you only need to make administrative changes (see the [Managing message deduplication](cookbooks-deduplication.md) cookbook for a guide). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.5.2/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.5.2/concepts-multi-tenancy.md index 8a17e72a4c8b3..26bd6a2fd183f 100644 --- a/site2/website/versioned_docs/version-2.5.2/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.5.2/concepts-multi-tenancy.md @@ -4,7 +4,7 @@ title: Multi Tenancy sidebar_label: "Multi Tenancy" --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -20,7 +20,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -28,7 +28,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.5.2/concepts-overview.md b/site2/website/versioned_docs/version-2.5.2/concepts-overview.md index c1262b38daf16..e8a2f4b9d321a 100644 --- a/site2/website/versioned_docs/version-2.5.2/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.5.2/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.5.2/concepts-replication.md b/site2/website/versioned_docs/version-2.5.2/concepts-replication.md index d4653ca7aecd8..9cc3a7c3415bd 100644 --- a/site2/website/versioned_docs/version-2.5.2/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.5.2/concepts-replication.md @@ -8,7 +8,7 @@ Regardless of industries, when an unforeseen event occurs and brings day-to-day Pulsar's geo-replication mechanism is typically used for disaster recovery, enabling the replication of persistently stored message data across multiple data centers. For instance, your application is publishing data in one region and you would like to process it for consumption in other regions. With Pulsar’s geo-replication mechanism, messages can be produced and consumed in different geo-locations. -The diagram below illustrates the process of [geo-replication](administration-geo). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. +The diagram below illustrates the process of [geo-replication](administration-geo.md). Whenever three producers (P1, P2 and P3) respectively publish messages to the T1 topic in three clusters, those messages are instantly replicated across clusters. Once the messages are replicated, two consumers (C1 and C2) can consume those messages from their clusters. ![A typical geo-replication example with full-mesh pattern](/assets/full-mesh-replication.svg) diff --git a/site2/website/versioned_docs/version-2.5.2/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.5.2/concepts-tiered-storage.md index f0bbde61718fb..ec031253ccd8d 100644 --- a/site2/website/versioned_docs/version-2.5.2/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.5.2/concepts-tiered-storage.md @@ -14,4 +14,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.5.2/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.5.2/concepts-topic-compaction.md index 790ce76b7c5e0..300cf68956dd1 100644 --- a/site2/website/versioned_docs/version-2.5.2/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.5.2/concepts-topic-compaction.md @@ -6,7 +6,7 @@ sidebar_label: "Topic Compaction" Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -23,7 +23,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.5.2/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.5.2/cookbooks-compaction.md index f436568e393de..864fb75255bfa 100644 --- a/site2/website/versioned_docs/version-2.5.2/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.5.2/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.5.2/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.5.2/cookbooks-non-persistent.md index d40c4fbd0cf01..1346d19a04263 100644 --- a/site2/website/versioned_docs/version-2.5.2/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.5.2/cookbooks-non-persistent.md @@ -40,7 +40,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -54,7 +54,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.5.2/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.5.2/cookbooks-partitioned.md index 31cc9cb8824c3..3f6ee52e9fc14 100644 --- a/site2/website/versioned_docs/version-2.5.2/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.5.2/cookbooks-partitioned.md @@ -7,7 +7,7 @@ original_id: cookbooks-partitioned By default, Pulsar topics are served by a single broker. Using only a single broker limits a topic's maximum throughput. *Partitioned topics* are a special type of topic that can span multiple brokers and thus allow for much higher throughput. For an explanation of how partitioned topics work, see the [Partitioned Topics](concepts-messaging.md#partitioned-topics) concepts. -You can publish to partitioned topics using Pulsar client libraries and you can [create and manage](#managing-partitioned-topics) partitioned topics using Pulsar [admin API](admin-api-overview). +You can publish to partitioned topics using Pulsar client libraries and you can [create and manage](#managing-partitioned-topics) partitioned topics using Pulsar [admin API](admin-api-overview.md). ## Publish to partitioned topics @@ -100,4 +100,4 @@ If a message has a key, it supersedes the round robin routing policy. The follow ## Manage partitioned topics -You can use Pulsar [admin API](admin-api-overview.md) to create and manage [partitioned topics](admin-api-partitioned-topics). +You can use Pulsar [admin API](admin-api-overview.md) to create and manage [partitioned topics](admin-api-partitioned-topics.md). diff --git a/site2/website/versioned_docs/version-2.5.2/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.5.2/cookbooks-retention-expiry.md index 86160c3f7ecf8..cd764fad6eb37 100644 --- a/site2/website/versioned_docs/version-2.5.2/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.5.2/cookbooks-retention-expiry.md @@ -21,7 +21,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.5.2/deploy-aws.md b/site2/website/versioned_docs/version-2.5.2/deploy-aws.md index 5db846fe790f2..60ac4c22d7672 100644 --- a/site2/website/versioned_docs/version-2.5.2/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.5.2/deploy-aws.md @@ -4,7 +4,7 @@ title: Deploying a Pulsar cluster on AWS using Terraform and Ansible sidebar_label: "Amazon Web Services" --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -209,7 +209,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.5.2/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.5.2/deploy-bare-metal-multi-cluster.md index 70dff45080b03..8a765d8843f43 100644 --- a/site2/website/versioned_docs/version-2.5.2/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.5.2/deploy-bare-metal-multi-cluster.md @@ -6,13 +6,13 @@ sidebar_label: "Bare metal multi-cluster" :::tip -1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster). -2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster.md). +2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo).Deploying a multi-cluster Pulsar instance consists of the following steps: +A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md).Deploying a multi-cluster Pulsar instance consists of the following steps: 1. Deploying two separate ZooKeeper quorums: a local quorum for each cluster in the instance and a configuration store quorum for instance-wide tasks 2. Initializing cluster metadata for each cluster @@ -21,7 +21,7 @@ A Pulsar instance consists of multiple Pulsar clusters working in unison. You ca > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. ## System requirement @@ -65,7 +65,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -244,7 +244,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -363,16 +363,16 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. -You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > **Service discovery already provided by many scheduling systems** -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.5.2/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.5.2/deploy-bare-metal.md index 4565eada114b7..fde30926519b1 100644 --- a/site2/website/versioned_docs/version-2.5.2/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.5.2/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using Pulsar in a startup or on a single team, it is simplest to opt for a single cluster. If you do need to run a multi-cluster Pulsar instance, -see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -263,9 +263,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (you had better not use a different port). -`--web-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). +`--web-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should not use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (you had better not use a different port). -`--broker-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). +`--broker-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). > If you do not have a DNS server, you can use multi-host format in the service URL with the following settings: @@ -389,7 +389,7 @@ webServicePortTls=8443 ### Enable Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -407,7 +407,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker). +If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker.md). ### Start Brokers diff --git a/site2/website/versioned_docs/version-2.5.2/deploy-dcos.md b/site2/website/versioned_docs/version-2.5.2/deploy-dcos.md index 07f446edaffd6..599d3a1b0454c 100644 --- a/site2/website/versioned_docs/version-2.5.2/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.5.2/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -To enable all built-in [Pulsar IO](io-overview) connectors in your Pulsar deploymente, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). +To enable all built-in [Pulsar IO](io-overview.md) connectors in your Pulsar deploymente, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.5.2/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.5.2/deploy-kubernetes.md index 4e170dcd4711d..7455247c901d4 100644 --- a/site2/website/versioned_docs/version-2.5.2/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.5.2/deploy-kubernetes.md @@ -5,6 +5,6 @@ sidebar_label: "Kubernetes" --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.5.2/deploy-monitoring.md b/site2/website/versioned_docs/version-2.5.2/deploy-monitoring.md index 2efc328097c00..40e918d9434fd 100644 --- a/site2/website/versioned_docs/version-2.5.2/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.5.2/deploy-monitoring.md @@ -73,7 +73,7 @@ The default port for bookie is `8000` (instead of `8080`). You can change the po You can use Prometheus to collect and store the metrics data. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes that needs to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is automatically setup with the [provided](deploy-kubernetes) instructions. +When you run Pulsar on bare metal, you can provide the list of nodes that needs to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is automatically setup with the [provided](deploy-kubernetes.md) instructions. ## Dashboards @@ -83,7 +83,7 @@ For that reason you only need to collect time series of metrics aggregated at th ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Dashboard](administration-dashboard). +The per-topic dashboard instructions are available at [Dashboard](administration-dashboard.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.5.2/develop-schema.md b/site2/website/versioned_docs/version-2.5.2/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.5.2/develop-schema.md +++ b/site2/website/versioned_docs/version-2.5.2/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.5.2/functions-deploy.md b/site2/website/versioned_docs/version-2.5.2/functions-deploy.md index aa4b799a0af61..a114758ee6bfe 100644 --- a/site2/website/versioned_docs/version-2.5.2/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.5.2/functions-deploy.md @@ -8,12 +8,12 @@ sidebar_label: "How-to: Deploy" To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), DC/OS, and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), DC/OS, and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface @@ -178,7 +178,7 @@ $ bin/pulsar-admin functions create \ Package management enables version management and simplifies the upgrade and rollback processes for Functions, Sinks, and Sources. When you use the same function, sink and source in different namespaces, you can upload them to a common package management system. -To use [Package management service](admin-api-packages), ensure that the package management service has been enabled in your cluster by setting the following properties in `broker.conf`. +To use [Package management service](admin-api-packages.md), ensure that the package management service has been enabled in your cluster by setting the following properties in `broker.conf`. > Note: Package management service is not enabled by default. diff --git a/site2/website/versioned_docs/version-2.5.2/functions-develop.md b/site2/website/versioned_docs/version-2.5.2/functions-develop.md index fc869e9362b7d..c61ee279d6cb7 100644 --- a/site2/website/versioned_docs/version-2.5.2/functions-develop.md +++ b/site2/website/versioned_docs/version-2.5.2/functions-develop.md @@ -854,7 +854,7 @@ public class MetricRecorderFunction implements Function { ``` -> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring) guide. +> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring.md) guide.     @@ -885,11 +885,11 @@ Currently, the feature is not available in Go. ```` ### Access metrics -To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring) in Pulsar. +To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring.md) in Pulsar. ## Security -If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker). For more details, refer to [Security settings](functions-worker.md#security-settings). +If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings). Pulsar Functions can support the following providers: diff --git a/site2/website/versioned_docs/version-2.5.2/functions-overview.md b/site2/website/versioned_docs/version-2.5.2/functions-overview.md index 25dc6029acf70..816d301e0fd0e 100644 --- a/site2/website/versioned_docs/version-2.5.2/functions-overview.md +++ b/site2/website/versioned_docs/version-2.5.2/functions-overview.md @@ -164,7 +164,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.5.2/functions-worker.md b/site2/website/versioned_docs/version-2.5.2/functions-worker.md index cfeab2b980c35..72ed20cc62c5d 100644 --- a/site2/website/versioned_docs/version-2.5.2/functions-worker.md +++ b/site2/website/versioned_docs/version-2.5.2/functions-worker.md @@ -3,7 +3,7 @@ id: functions-worker title: Deploy and manage functions worker sidebar_label: "Setup: Pulsar Functions Worker" --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -198,7 +198,7 @@ brokerClientTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -218,7 +218,7 @@ authenticationProviders: [ provider1, provider2 ] ``` For *TLS Authentication* provider, follow the example below to add the necessary settings. -See [TLS Authentication](security-tls-authentication) for more details. +See [TLS Authentication](security-tls-authentication.md) for more details. ``` @@ -242,7 +242,7 @@ properties: ``` For *Token Authentication* provider, add necessary settings for `properties` if needed. -See [Token Authentication](security-jwt) for more details. +See [Token Authentication](security-jwt.md) for more details. Note: key files must be DER-encoded ``` @@ -281,7 +281,7 @@ superUserRoles: You can use the public and private key pair that the application configures to perform encryption. Only the consumers with a valid key can decrypt the encrypted messages. -To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption). +To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption.md). We include the relevant configuration information of `CryptoConfig` into `ProducerConfig`. The specific configurable field information about `CryptoConfig` is as follows: diff --git a/site2/website/versioned_docs/version-2.5.2/getting-started-docker.md b/site2/website/versioned_docs/version-2.5.2/getting-started-docker.md index 26908245cb2c9..904d67401668a 100644 --- a/site2/website/versioned_docs/version-2.5.2/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.5.2/getting-started-docker.md @@ -43,8 +43,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -103,7 +103,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.5.2/getting-started-helm.md b/site2/website/versioned_docs/version-2.5.2/getting-started-helm.md index 4dab89971de09..b52f59593c994 100644 --- a/site2/website/versioned_docs/version-2.5.2/getting-started-helm.md +++ b/site2/website/versioned_docs/version-2.5.2/getting-started-helm.md @@ -12,7 +12,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -28,7 +28,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -405,7 +405,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.5.2/getting-started-standalone.md b/site2/website/versioned_docs/version-2.5.2/getting-started-standalone.md index b8fe606a581cd..adcd70ab7da90 100644 --- a/site2/website/versioned_docs/version-2.5.2/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.5.2/getting-started-standalone.md @@ -7,7 +7,7 @@ sidebar_label: "Run Pulsar locally" For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary [RocksDB](http://rocksdb.org/) and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > **Pulsar in production?** -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -64,9 +64,9 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). -`conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker) and more.
    **Note:** Pulsar standalone uses RocksDB as the local metadata store and its configuration file path [`metadataStoreConfigPath`](reference-configuration) is configurable in the `standalone.conf` file. For more information about the configurations of RocksDB, see [here](https://github.com/facebook/rocksdb/blob/main/examples/rocksdb_option_file_example.ini) and related [documentation](https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide). -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker) and more.
    **Note:** Pulsar standalone uses RocksDB as the local metadata store and its configuration file path [`metadataStoreConfigPath`](reference-configuration.md) is configurable in the `standalone.conf` file. For more information about the configurations of RocksDB, see [here](https://github.com/facebook/rocksdb/blob/main/examples/rocksdb_option_file_example.ini) and related [documentation](https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide). +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -122,7 +122,7 @@ pulsar-io-aerospike-@pulsar:version@.nar :::note * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -135,7 +135,7 @@ pulsar-io-aerospike-@pulsar:version@.nar ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -168,12 +168,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or DC/OS), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or DC/OS), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -205,7 +205,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `bin/pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.5.2/helm-install.md b/site2/website/versioned_docs/version-2.5.2/helm-install.md index dd9c8afdad185..16a646d9aa56f 100644 --- a/site2/website/versioned_docs/version-2.5.2/helm-install.md +++ b/site2/website/versioned_docs/version-2.5.2/helm-install.md @@ -20,7 +20,7 @@ Before deploying Pulsar, you need to prepare your environment. ### Tools -Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools) on your computer. +Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools.md) on your computer. ## Cloud cluster preparation @@ -30,8 +30,8 @@ To create and connect to the Kubernetes cluster, follow the instructions: ## Pulsar deployment -Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy). +Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy.md). ## Pulsar upgrade -To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade). +To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade.md). diff --git a/site2/website/versioned_docs/version-2.5.2/helm-overview.md b/site2/website/versioned_docs/version-2.5.2/helm-overview.md index 9afa1533bbab4..9aa05859c0449 100644 --- a/site2/website/versioned_docs/version-2.5.2/helm-overview.md +++ b/site2/website/versioned_docs/version-2.5.2/helm-overview.md @@ -51,9 +51,9 @@ Moreover, Helm chart supports: ## Quick start -To run with Apache Pulsar Helm chart as fast as possible in a **non-production** use case, we provide a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +To run with Apache Pulsar Helm chart as fast as possible in a **non-production** use case, we provide a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -This guide walks you through deploying Apache Pulsar Helm chart with default values and features, but it is *not* suitable for deployments in production-ready environments. To deploy the charts in production under sustained load, you can follow the complete [Installation Guide](helm-install). +This guide walks you through deploying Apache Pulsar Helm chart with default values and features, but it is *not* suitable for deployments in production-ready environments. To deploy the charts in production under sustained load, you can follow the complete [Installation Guide](helm-install.md). ## Troubleshooting @@ -63,7 +63,7 @@ Although we have done our best to make these charts as seamless as possible, tro The Apache Pulsar Helm chart contains all required dependencies. -If you deploy a PoC for testing, we strongly suggest you follow this [Quick Start Guide](getting-started-helm) for your first iteration. +If you deploy a PoC for testing, we strongly suggest you follow this [Quick Start Guide](getting-started-helm.md) for your first iteration. 1. [Preparation](helm-prepare.md) 2. [Deployment](helm-deploy.md) @@ -81,7 +81,7 @@ helm upgrade apache/pulsar -f pulsar.yaml ``` -For more detailed information, see [Upgrading](helm-upgrade). +For more detailed information, see [Upgrading](helm-upgrade.md). ## Uninstallation diff --git a/site2/website/versioned_docs/version-2.5.2/helm-tools.md b/site2/website/versioned_docs/version-2.5.2/helm-tools.md index 7a9527160a131..c594990176407 100644 --- a/site2/website/versioned_docs/version-2.5.2/helm-tools.md +++ b/site2/website/versioned_docs/version-2.5.2/helm-tools.md @@ -24,7 +24,7 @@ You can get Helm from the project's [releases page](https://github.com/helm/helm ### Next steps -Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare). +Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare.md). ## Additional information diff --git a/site2/website/versioned_docs/version-2.5.2/helm-upgrade.md b/site2/website/versioned_docs/version-2.5.2/helm-upgrade.md index 9713557f8ba24..71b2ac4a7f6d4 100644 --- a/site2/website/versioned_docs/version-2.5.2/helm-upgrade.md +++ b/site2/website/versioned_docs/version-2.5.2/helm-upgrade.md @@ -19,7 +19,7 @@ You can retrieve your previous `--set` arguments cleanly, with `helm get values To upgrade Apache Pulsar to a newer version, follow these steps: 1. Check the change log for the specific version you would like to upgrade to. -2. Go through [deployment documentation](helm-deploy) step by step. +2. Go through [deployment documentation](helm-deploy.md) step by step. 3. Extract your previous `--set` arguments with the following command. ```bash diff --git a/site2/website/versioned_docs/version-2.5.2/io-cdc.md b/site2/website/versioned_docs/version-2.5.2/io-cdc.md index df373977e3eaa..a3527dc0a9d65 100644 --- a/site2/website/versioned_docs/version-2.5.2/io-cdc.md +++ b/site2/website/versioned_docs/version-2.5.2/io-cdc.md @@ -12,8 +12,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.5.2/io-develop.md b/site2/website/versioned_docs/version-2.5.2/io-develop.md index 4a815b3b9d6f6..7a356662560c9 100644 --- a/site2/website/versioned_docs/version-2.5.2/io-develop.md +++ b/site2/website/versioned_docs/version-2.5.2/io-develop.md @@ -8,15 +8,15 @@ original_id: io-develop This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -163,7 +163,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -192,7 +192,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). diff --git a/site2/website/versioned_docs/version-2.5.2/io-overview.md b/site2/website/versioned_docs/version-2.5.2/io-overview.md index 2a6ecaee5883f..ea49e4b042e8c 100644 --- a/site2/website/versioned_docs/version-2.5.2/io-overview.md +++ b/site2/website/versioned_docs/version-2.5.2/io-overview.md @@ -157,7 +157,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.5.2/io-quickstart.md b/site2/website/versioned_docs/version-2.5.2/io-quickstart.md index 0c9c78caec8a0..6ebf49d0f88b7 100644 --- a/site2/website/versioned_docs/version-2.5.2/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.5.2/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you are able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you are able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However, all the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -109,7 +109,7 @@ This section demonstrates how to connector Pulsar to Cassandra. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink.md). ::: @@ -236,11 +236,11 @@ You can create a configuration file through one of the following methods. ``` -For more information, see [Cassandra sink connector](io-cassandra-sink). +For more information, see [Cassandra sink connector](io-cassandra-sink.md). ### Create a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on them. Run the following command to create a Cassandra sink connector with sink type _cassandra_ and the config file _examples/cassandra-sink.yml_ created previously. @@ -268,7 +268,7 @@ as a Pulsar Function and writes the messages produced in the topic _test_cassand ### Inspect a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * Get the information of a Cassandra sink. @@ -432,7 +432,7 @@ to monitor a connector and perform other operations on it. ### Delete a Cassandra Sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. ```bash @@ -451,7 +451,7 @@ This section demonstrates how to connector Pulsar to MySQL. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The JDBC sink connector pulls messages from Pulsar topics and persists the messages to MySQL or SQlite. For more information, see [JDBC sink connector](io-jdbc-sink). +* The JDBC sink connector pulls messages from Pulsar topics and persists the messages to MySQL or SQlite. For more information, see [JDBC sink connector](io-jdbc-sink.md). ::: @@ -624,7 +624,7 @@ In this section, you need to configure a JDBC sink connector. ### Create a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on it. This example creates a sink connector and specifies the desired information. @@ -670,7 +670,7 @@ The sink has been created successfully if the following message appears. ### Inspect a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * List all running JDBC sink(s). @@ -790,7 +790,7 @@ to monitor a connector and perform other operations on it. ### Stop a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to stop a connector and perform other operations on it. ```bash @@ -819,7 +819,7 @@ The sink instance has been stopped successfully if the following message disappe ### Restart a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to restart a connector and perform other operations on it. ```bash @@ -856,7 +856,7 @@ Note that `pulsar-admin sinks localrun options` **runs a sink connector locally* ### Update a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to update a connector and perform other operations on it. This example updates the parallelism of the _pulsar-mysql-jdbc-sink_ sink connector to 2. @@ -924,7 +924,7 @@ The result shows that the parallelism is 2. ### Delete a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. This example deletes the _pulsar-mysql-jdbc-sink_ sink connector. diff --git a/site2/website/versioned_docs/version-2.5.2/io-use.md b/site2/website/versioned_docs/version-2.5.2/io-use.md index d8c5aaa5c8d88..6760a5f61a125 100644 --- a/site2/website/versioned_docs/version-2.5.2/io-use.md +++ b/site2/website/versioned_docs/version-2.5.2/io-use.md @@ -14,7 +14,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.5.2/kubernetes-helm.md b/site2/website/versioned_docs/version-2.5.2/kubernetes-helm.md index a9f768e6a96b3..ea92a0968cd7d 100644 --- a/site2/website/versioned_docs/version-2.5.2/kubernetes-helm.md +++ b/site2/website/versioned_docs/version-2.5.2/kubernetes-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.5.2/pulsar-2.0.md b/site2/website/versioned_docs/version-2.5.2/pulsar-2.0.md index 560c8c1bd7cb7..07fba365e9ff2 100644 --- a/site2/website/versioned_docs/version-2.5.2/pulsar-2.0.md +++ b/site2/website/versioned_docs/version-2.5.2/pulsar-2.0.md @@ -4,13 +4,13 @@ title: Pulsar 2.0 sidebar_label: "Pulsar 2.0" --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.5.2/reference-cli-tools.md b/site2/website/versioned_docs/version-2.5.2/reference-cli-tools.md index 8516541f10cca..9e21a017071cd 100644 --- a/site2/website/versioned_docs/version-2.5.2/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.5.2/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.5.2/schema-get-started.md b/site2/website/versioned_docs/version-2.5.2/schema-get-started.md index 97e270586a26d..afacb0fa51f2e 100644 --- a/site2/website/versioned_docs/version-2.5.2/schema-get-started.md +++ b/site2/website/versioned_docs/version-2.5.2/schema-get-started.md @@ -17,7 +17,7 @@ Applications typically adopt one of the following approaches to guarantee type s #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ### Client-side approach diff --git a/site2/website/versioned_docs/version-2.5.2/schema-manage.md b/site2/website/versioned_docs/version-2.5.2/schema-manage.md index 072078dd5828d..7a3e3d75366f8 100644 --- a/site2/website/versioned_docs/version-2.5.2/schema-manage.md +++ b/site2/website/versioned_docs/version-2.5.2/schema-manage.md @@ -159,7 +159,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.5.2/schema-understand.md b/site2/website/versioned_docs/version-2.5.2/schema-understand.md index fda3d6dfecb4c..2b5fd1f44b360 100644 --- a/site2/website/versioned_docs/version-2.5.2/schema-understand.md +++ b/site2/website/versioned_docs/version-2.5.2/schema-understand.md @@ -372,7 +372,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```java diff --git a/site2/website/versioned_docs/version-2.5.2/security-athenz.md b/site2/website/versioned_docs/version-2.5.2/security-athenz.md index ba27ba43eb9d8..6d338fc4013bd 100644 --- a/site2/website/versioned_docs/version-2.5.2/security-athenz.md +++ b/site2/website/versioned_docs/version-2.5.2/security-athenz.md @@ -75,7 +75,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.5.2/security-authorization.md b/site2/website/versioned_docs/version-2.5.2/security-authorization.md index d9d33665d066b..c5e3c04b03524 100644 --- a/site2/website/versioned_docs/version-2.5.2/security-authorization.md +++ b/site2/website/versioned_docs/version-2.5.2/security-authorization.md @@ -26,7 +26,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -58,7 +58,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -86,7 +86,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication @@ -116,7 +116,7 @@ PulsarAdmin admin = PulsarAdmin.builder() When a client is identified with multiple roles in a token (the type of role claim in the token is an array) during the authentication process, Pulsar supports to check the permissions of all the roles and further authorize the client as long as one of its roles has the required permissions. > **Note**
    -> This authorization method is only compatible with [JWT authentication](security-jwt). +> This authorization method is only compatible with [JWT authentication](security-jwt.md). To enable this authorization method, configure the authorization provider as `MultiRolesTokenAuthorizationProvider` in the `conf/broker.conf` file. diff --git a/site2/website/versioned_docs/version-2.5.2/security-bouncy-castle.md b/site2/website/versioned_docs/version-2.5.2/security-bouncy-castle.md index 1012f0e4933b6..eaf4f907f17e4 100644 --- a/site2/website/versioned_docs/version-2.5.2/security-bouncy-castle.md +++ b/site2/website/versioned_docs/version-2.5.2/security-bouncy-castle.md @@ -16,7 +16,7 @@ In Pulsar, security and crypto have dependencies on BouncyCastle Jars. For the d `Bouncy Castle` provides both [FIPS](https://www.bouncycastle.org/fips_faq.html) and non-FIPS version. But in a JVM, you can not include both of the 2 versions, and you need to exclude the current version before include the other. -In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. +In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption.md). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. ## Include dependencies of BC-non-FIPS diff --git a/site2/website/versioned_docs/version-2.5.2/security-jwt.md b/site2/website/versioned_docs/version-2.5.2/security-jwt.md index e702147251f8f..ef032985114bd 100644 --- a/site2/website/versioned_docs/version-2.5.2/security-jwt.md +++ b/site2/website/versioned_docs/version-2.5.2/security-jwt.md @@ -32,11 +32,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -288,7 +288,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.5.2/security-kerberos.md b/site2/website/versioned_docs/version-2.5.2/security-kerberos.md index c3d6fbe044889..9d77149e845e3 100644 --- a/site2/website/versioned_docs/version-2.5.2/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.5.2/security-kerberos.md @@ -372,7 +372,7 @@ saslJaasServerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.5.2/security-overview.md b/site2/website/versioned_docs/version-2.5.2/security-overview.md index 2f3fc035ad4cb..3f08cb29ad854 100644 --- a/site2/website/versioned_docs/version-2.5.2/security-overview.md +++ b/site2/website/versioned_docs/version-2.5.2/security-overview.md @@ -20,7 +20,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.5.2/security-tls-authentication.md b/site2/website/versioned_docs/version-2.5.2/security-tls-authentication.md index 2987325ec3221..3a3e35b731000 100644 --- a/site2/website/versioned_docs/version-2.5.2/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.5.2/security-tls-authentication.md @@ -7,9 +7,9 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -101,7 +101,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -121,7 +121,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.5.2/security-tls-transport.md b/site2/website/versioned_docs/version-2.5.2/security-tls-transport.md index a1a9f3955daaf..9fbbe2bae2cab 100644 --- a/site2/website/versioned_docs/version-2.5.2/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.5.2/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,9 +27,9 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). -`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ## Create TLS certificates @@ -130,7 +130,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Broker Configuration -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -201,7 +201,7 @@ The examples below show hostname verification being disabled for the Java client ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.5.2/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.5.2/sql-deployment-configurations.md index 639ac988b3409..069c6ac9c92da 100644 --- a/site2/website/versioned_docs/version-2.5.2/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.5.2/sql-deployment-configurations.md @@ -141,7 +141,7 @@ Since Pulsar SQL is powered by [Trino (formerly Presto SQL)](https://trino.io), :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.5.2/sql-overview.md b/site2/website/versioned_docs/version-2.5.2/sql-overview.md index 2f827f4267933..5b01dfde56e7f 100644 --- a/site2/website/versioned_docs/version-2.5.2/sql-overview.md +++ b/site2/website/versioned_docs/version-2.5.2/sql-overview.md @@ -4,7 +4,7 @@ title: Pulsar SQL Overview sidebar_label: "Overview" --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL)](https://trino.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL.md)](https://trino.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.5.2/standalone-docker.md b/site2/website/versioned_docs/version-2.5.2/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.5.2/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.5.2/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.5.2/standalone.md b/site2/website/versioned_docs/version-2.5.2/standalone.md index 233394260c050..cfbd7719e5e6e 100644 --- a/site2/website/versioned_docs/version-2.5.2/standalone.md +++ b/site2/website/versioned_docs/version-2.5.2/standalone.md @@ -7,7 +7,7 @@ sidebar_label: "Run Pulsar locally" For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary [RocksDB](http://rocksdb.org/) and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > **Pulsar in production?** -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -63,9 +63,9 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). -`conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker) and more.
    **Note:** Pulsar standalone uses RocksDB as the local metadata store and its configuration file path [`metadataStoreConfigPath`](reference-configuration) is configurable in the `standalone.conf` file. For more information about the configurations of RocksDB, see [here](https://github.com/facebook/rocksdb/blob/main/examples/rocksdb_option_file_example.ini) and related [documentation](https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide). -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker) and more.
    **Note:** Pulsar standalone uses RocksDB as the local metadata store and its configuration file path [`metadataStoreConfigPath`](reference-configuration.md) is configurable in the `standalone.conf` file. For more information about the configurations of RocksDB, see [here](https://github.com/facebook/rocksdb/blob/main/examples/rocksdb_option_file_example.ini) and related [documentation](https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide). +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -121,7 +121,7 @@ pulsar-io-aerospike-@pulsar:version@.nar :::note * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -134,7 +134,7 @@ pulsar-io-aerospike-@pulsar:version@.nar ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -167,12 +167,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or DC/OS), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or DC/OS), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -204,7 +204,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `bin/pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.6.0/admin-api-brokers.md b/site2/website/versioned_docs/version-2.6.0/admin-api-brokers.md index bdd6ff555ac74..9db8f153bf5da 100644 --- a/site2/website/versioned_docs/version-2.6.0/admin-api-brokers.md +++ b/site2/website/versioned_docs/version-2.6.0/admin-api-brokers.md @@ -12,7 +12,7 @@ Pulsar brokers consist of two components: [Brokers](reference-terminology.md#broker) can be managed via: -* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/brokers` endpoint of the admin {@inject: rest:REST:/} API * The `brokers` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin.html} object in the [Java API](client-libraries-java.md) @@ -95,7 +95,7 @@ One way to configure a Pulsar [broker](reference-terminology.md#broker) is to su But since all broker configuration in Pulsar is stored in ZooKeeper, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values. -* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. +* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. * In the Pulsar admin {@inject: rest:REST:/} API, dynamic configuration is managed through the `/admin/v2/brokers/configuration` endpoint. ### Update dynamic configuration diff --git a/site2/website/versioned_docs/version-2.6.0/admin-api-clusters.md b/site2/website/versioned_docs/version-2.6.0/admin-api-clusters.md index efd50a4c4b0c4..ac489a949d0f2 100644 --- a/site2/website/versioned_docs/version-2.6.0/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.6.0/admin-api-clusters.md @@ -10,7 +10,7 @@ servers (aka [bookies](reference-terminology.md#bookie)), and a [ZooKeeper](http Clusters can be managed via: -* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/clusters` endpoint of the admin {@inject: rest:REST:/} API * The `clusters` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md) @@ -62,7 +62,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > #### No cluster metadata initialization through the REST API or the Java admin API > @@ -86,11 +86,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. #### pulsar-admin diff --git a/site2/website/versioned_docs/version-2.6.0/admin-api-functions.md b/site2/website/versioned_docs/version-2.6.0/admin-api-functions.md index 96b7dc7ea52cd..315943492c214 100644 --- a/site2/website/versioned_docs/version-2.6.0/admin-api-functions.md +++ b/site2/website/versioned_docs/version-2.6.0/admin-api-functions.md @@ -15,9 +15,9 @@ Functions can be managed via the following methods. Method | Description ---|--- -**Admin CLI** | The [`functions`](reference-pulsar-admin.md#functions) command of the [`pulsar-admin`](reference-pulsar-admin) tool. +**Admin CLI** | The [`functions`](reference-pulsar-admin.md#functions) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool. **REST API** |The `/admin/v3/functions` endpoint of the admin {@inject: rest:REST:/} API. -**Java Admin API**| The `functions` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java). +**Java Admin API**| The `functions` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md). ## Function resources diff --git a/site2/website/versioned_docs/version-2.6.0/admin-api-namespaces.md b/site2/website/versioned_docs/version-2.6.0/admin-api-namespaces.md index c73dbcfd3f24d..c3c4461b48bd0 100644 --- a/site2/website/versioned_docs/version-2.6.0/admin-api-namespaces.md +++ b/site2/website/versioned_docs/version-2.6.0/admin-api-namespaces.md @@ -9,7 +9,7 @@ Pulsar [namespaces](reference-terminology.md#namespace) are logical groupings of Namespaces can be managed via: -* The [`namespaces`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`namespaces`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/namespaces` endpoint of the admin {@inject: rest:REST:/} API * The `namespaces` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md) diff --git a/site2/website/versioned_docs/version-2.6.0/admin-api-non-partitioned-topics.md b/site2/website/versioned_docs/version-2.6.0/admin-api-non-partitioned-topics.md index 9cd019d2abe8f..d34d8def6473f 100644 --- a/site2/website/versioned_docs/version-2.6.0/admin-api-non-partitioned-topics.md +++ b/site2/website/versioned_docs/version-2.6.0/admin-api-non-partitioned-topics.md @@ -6,7 +6,7 @@ original_id: admin-api-non-partitioned-topics --- -You can use Pulsar's [admin API](admin-api-overview) to create and manage non-partitioned topics. +You can use Pulsar's [admin API](admin-api-overview.md) to create and manage non-partitioned topics. In all of the instructions and commands below, the topic name structure is: diff --git a/site2/website/versioned_docs/version-2.6.0/admin-api-overview.md b/site2/website/versioned_docs/version-2.6.0/admin-api-overview.md index b6186cdd0ea9a..ec19fceda0db2 100644 --- a/site2/website/versioned_docs/version-2.6.0/admin-api-overview.md +++ b/site2/website/versioned_docs/version-2.6.0/admin-api-overview.md @@ -12,7 +12,7 @@ You can currently interact with the admin interface via: - Making HTTP calls against the admin {@inject: rest:REST:/} API provided by Pulsar [brokers](reference-terminology.md#broker). For some restful apis, they might be redirected to topic owner brokers for serving with [`307 Temporary Redirect`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/307), hence the HTTP callers should handle `307 Temporary Redirect`. If you are using `curl`, you should specify `-L` to handle redirections. -- The `pulsar-admin` CLI tool, which is available in the `bin` folder of your [Pulsar installation](getting-started-standalone): +- The `pulsar-admin` CLI tool, which is available in the `bin` folder of your [Pulsar installation](getting-started-standalone.md): ```shell @@ -20,7 +20,7 @@ $ bin/pulsar-admin ``` -Full documentation for this tool can be found in the [Pulsar command-line tools](reference-pulsar-admin) doc. +Full documentation for this tool can be found in the [Pulsar command-line tools](reference-pulsar-admin.md) doc. - A Java client interface. @@ -31,11 +31,11 @@ In this document, examples from each of the three available interfaces will be s ## Admin setup -Each of Pulsar's three admin interfaces---the [`pulsar-admin`](reference-pulsar-admin) CLI tool, the [Java admin API](/api/admin), and the {@inject: rest:REST:/} API ---requires some special setup if you have [authentication](security-overview.md#authentication-providers) enabled in your Pulsar [instance](reference-terminology.md#instance). +Each of Pulsar's three admin interfaces---the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool, the [Java admin API](/api/admin), and the {@inject: rest:REST:/} API ---requires some special setup if you have [authentication](security-overview.md#authentication-providers) enabled in your Pulsar [instance](reference-terminology.md#instance). ### pulsar-admin -If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: +If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin.md) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: |Name|Description|Default| |----|-----------|-------| diff --git a/site2/website/versioned_docs/version-2.6.0/admin-api-partitioned-topics.md b/site2/website/versioned_docs/version-2.6.0/admin-api-partitioned-topics.md index eef6efdf7407b..73edab60a4039 100644 --- a/site2/website/versioned_docs/version-2.6.0/admin-api-partitioned-topics.md +++ b/site2/website/versioned_docs/version-2.6.0/admin-api-partitioned-topics.md @@ -6,7 +6,7 @@ original_id: admin-api-partitioned-topics --- -You can use Pulsar's [admin API](admin-api-overview) to create and manage partitioned topics. +You can use Pulsar's [admin API](admin-api-overview.md) to create and manage partitioned topics. In all of the instructions and commands below, the topic name structure is: diff --git a/site2/website/versioned_docs/version-2.6.0/admin-api-tenants.md b/site2/website/versioned_docs/version-2.6.0/admin-api-tenants.md index 924cbce3d41ab..27764d6da3885 100644 --- a/site2/website/versioned_docs/version-2.6.0/admin-api-tenants.md +++ b/site2/website/versioned_docs/version-2.6.0/admin-api-tenants.md @@ -5,7 +5,7 @@ sidebar_label: "Tenants" original_id: admin-api-tenants --- -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters diff --git a/site2/website/versioned_docs/version-2.6.0/administration-dashboard.md b/site2/website/versioned_docs/version-2.6.0/administration-dashboard.md index 514b076239789..25f976609b40b 100644 --- a/site2/website/versioned_docs/version-2.6.0/administration-dashboard.md +++ b/site2/website/versioned_docs/version-2.6.0/administration-dashboard.md @@ -7,7 +7,7 @@ original_id: administration-dashboard :::note -Pulsar dashboard is deprecated. If you want to manage and monitor the stats of your topics, use [Pulsar Manager](administration-pulsar-manager). +Pulsar dashboard is deprecated. If you want to manage and monitor the stats of your topics, use [Pulsar Manager](administration-pulsar-manager.md). ::: diff --git a/site2/website/versioned_docs/version-2.6.0/administration-geo.md b/site2/website/versioned_docs/version-2.6.0/administration-geo.md index 42d32304a2fa4..4170d62f3ac9c 100644 --- a/site2/website/versioned_docs/version-2.6.0/administration-geo.md +++ b/site2/website/versioned_docs/version-2.6.0/administration-geo.md @@ -135,7 +135,7 @@ Once you create a geo-replication namespace, any topics that producers or consum By default, messages are replicated to all clusters configured for the namespace. You can restrict replication selectively by specifying a replication list for a message, and then that message is replicated only to the subset in the replication list. -The following is an example for the [Java API](client-libraries-java). Note the use of the `setReplicationClusters` method when you construct the {@inject: javadoc:Message:/client/org/apache/pulsar/client/api/Message} object: +The following is an example for the [Java API](client-libraries-java.md). Note the use of the `setReplicationClusters` method when you construct the {@inject: javadoc:Message:/client/org/apache/pulsar/client/api/Message} object: ```java @@ -157,7 +157,7 @@ producer.newMessage() #### Topic stats -Topic-specific statistics for geo-replication topics are available via the [`pulsar-admin`](reference-pulsar-admin) tool and {@inject: rest:REST:/} API: +Topic-specific statistics for geo-replication topics are available via the [`pulsar-admin`](reference-pulsar-admin.md) tool and {@inject: rest:REST:/} API: ```shell diff --git a/site2/website/versioned_docs/version-2.6.0/administration-zk-bk.md b/site2/website/versioned_docs/version-2.6.0/administration-zk-bk.md index 1bbf990bd1d4d..42019d74afba3 100644 --- a/site2/website/versioned_docs/version-2.6.0/administration-zk-bk.md +++ b/site2/website/versioned_docs/version-2.6.0/administration-zk-bk.md @@ -174,7 +174,7 @@ The [`conf/global-zookeeper.conf`](reference-configuration.md#configuration-stor BookKeeper stores all durable messages in Pulsar. BookKeeper is a distributed [write-ahead log](https://en.wikipedia.org/wiki/Write-ahead_logging) WAL system that guarantees read consistency of independent message logs calls ledgers. Individual BookKeeper servers are also called *bookies*. -> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.6.0/client-libraries-cgo.md b/site2/website/versioned_docs/version-2.6.0/client-libraries-cgo.md index c79f7bb965473..f352f942b7714 100644 --- a/site2/website/versioned_docs/version-2.6.0/client-libraries-cgo.md +++ b/site2/website/versioned_docs/version-2.6.0/client-libraries-cgo.md @@ -24,7 +24,7 @@ Currently, the following Go clients are maintained in two repositories. ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Install go package @@ -57,7 +57,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -75,7 +75,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -236,14 +236,14 @@ Parameter | Description | Default `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | `Properties`| Attach a set of application defined properties to the producer. This properties will be visible in the topic stats | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | Set the number of max pending messages across all the partitions. This setting will be used to lower the max pending messages for each partition `MaxPendingMessages(int)`, if the total exceeds the configured value.| `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | `Batching` | Control whether automatic batching of messages is enabled for the producer. | false `BatchingMaxPublishDelay` | Set the time period within which the messages sent will be batched (default: 1ms) if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or until | 1ms `BatchingMaxMessages` | Set the maximum number of messages permitted in a batch. (default: 1000) If set to a value greater than 1, messages will be queued until this threshold is reached or batch interval has elapsed | 1000 @@ -512,7 +512,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.6.0/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.6.0/client-libraries-cpp.md index bb52d9effa404..684df5fd2d825 100644 --- a/site2/website/versioned_docs/version-2.6.0/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.6.0/client-libraries-cpp.md @@ -261,7 +261,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema -This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started). +This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started.md). ### Create producer with Avro schema diff --git a/site2/website/versioned_docs/version-2.6.0/client-libraries-dotnet.md b/site2/website/versioned_docs/version-2.6.0/client-libraries-dotnet.md index 4e0afe3d2ff6b..fbec5e473be69 100644 --- a/site2/website/versioned_docs/version-2.6.0/client-libraries-dotnet.md +++ b/site2/website/versioned_docs/version-2.6.0/client-libraries-dotnet.md @@ -170,7 +170,7 @@ var client = PulsarClient.Builder() Currently, the Pulsar C# client supports the TLS (Transport Layer Security) and JWT (JSON Web Token) authentication. -If you have followed [Authentication using TLS](security-tls-authentication), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: +If you have followed [Authentication using TLS](security-tls-authentication.md), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: 1. Create an unencrypted and password-less pfx file. diff --git a/site2/website/versioned_docs/version-2.6.0/client-libraries-go.md b/site2/website/versioned_docs/version-2.6.0/client-libraries-go.md index f32f308aeb4b5..350e8db2e92c5 100644 --- a/site2/website/versioned_docs/version-2.6.0/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.6.0/client-libraries-go.md @@ -35,7 +35,7 @@ import "github.com/apache/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -53,7 +53,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -680,7 +680,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.6.0/client-libraries-java.md b/site2/website/versioned_docs/version-2.6.0/client-libraries-java.md index e53110f93c77e..4b2aaae5dcc38 100644 --- a/site2/website/versioned_docs/version-2.6.0/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.6.0/client-libraries-java.md @@ -5,7 +5,7 @@ sidebar_label: "Java" original_id: client-libraries-java --- -You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), and [reader](#reader) of a Java client are thread-safe. @@ -14,10 +14,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Includes both `pulsar-client` and `pulsar-client-admin`

    Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.

    In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -57,7 +57,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -83,7 +83,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -114,7 +114,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -237,7 +237,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -801,7 +801,7 @@ Total hash range size is 65536, so the max end of the range should be less than ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -915,7 +915,7 @@ The following schema formats are currently available for Java: ## Authentication -Pulsar currently supports two authentication schemes: [TLS](security-tls-authentication.md) and [Athenz](security-athenz). You can use the Pulsar Java client with both. +Pulsar currently supports two authentication schemes: [TLS](security-tls-authentication.md) and [Athenz](security-athenz.md). You can use the Pulsar Java client with both. ### TLS Authentication @@ -942,7 +942,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` diff --git a/site2/website/versioned_docs/version-2.6.0/client-libraries-node.md b/site2/website/versioned_docs/version-2.6.0/client-libraries-node.md index 8d7376d3793e5..b5a138cb86528 100644 --- a/site2/website/versioned_docs/version-2.6.0/client-libraries-node.md +++ b/site2/website/versioned_docs/version-2.6.0/client-libraries-node.md @@ -44,7 +44,7 @@ Also, this library works only in Node.js 10.x or later because it uses the [`nod ::: ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here is an example for `localhost`: @@ -62,7 +62,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication), the URL will look like something like this: +If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication.md), the URL will look like something like this: ```http @@ -97,7 +97,7 @@ The following configurable parameters are available for Pulsar clients: | Parameter | Description | Default | | :-------- | :---------- | :------ | | `serviceUrl` | The connection URL for the Pulsar cluster. See [above](#connection-urls) for more info. | | -| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication) for more info. | | +| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication.md) for more info. | | | `operationTimeoutSeconds` | The timeout for Node.js client operations (creating producers, subscribing to and unsubscribing from [topics](reference-terminology.md#topic)). Retries will occur until this threshold is reached, at which point the operation will fail. | 30 | | `ioThreads` | The number of threads to use for handling connections to Pulsar [brokers](reference-terminology.md#broker). | 1 | | `messageListenerThreads` | The number of threads used by message listeners ([consumers](#consumers) and [readers](#readers)). | 1 | @@ -147,7 +147,7 @@ Pulsar Node.js producers have the following methods available: | :-------- | :---------- | :------ | | `topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer publishes messages. The topic format is `` or `//`. For example, `sample/ns1/my-topic`. | | | `producerName` | A name for the producer. If you do not explicitly assign a name, Pulsar will automatically generate a globally unique name. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | | -| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30000 | +| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30000 | | `initialSequenceId` | The initial sequence ID of the message. When producer send message, add sequence ID to message. The ID is increased each time to send. | | | `maxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `send` method will fail *unless* `blockIfQueueFull` is set to `true`. | 1000 | | `maxPendingMessagesAcrossPartitions` | The maximum size of the sum of partition's pending queue. | 50000 | diff --git a/site2/website/versioned_docs/version-2.6.0/client-libraries-python.md b/site2/website/versioned_docs/version-2.6.0/client-libraries-python.md index f9f9392ea68eb..5435a6a99a046 100644 --- a/site2/website/versioned_docs/version-2.6.0/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.6.0/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. diff --git a/site2/website/versioned_docs/version-2.6.0/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.6.0/client-libraries-websocket.md index 9cc5d4acb8ded..f939a32d19b13 100644 --- a/site2/website/versioned_docs/version-2.6.0/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.6.0/client-libraries-websocket.md @@ -5,14 +5,14 @@ sidebar_label: "WebSocket" original_id: client-libraries-websocket --- -Pulsar's [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API is meant to provide a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSockets you can publish and consume messages and use all the features available in the [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp) client libraries. +Pulsar's [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API is meant to provide a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSockets you can publish and consume messages and use all the features available in the [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md) client libraries. > You can use Pulsar's WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.6.0/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.6.0/concepts-architecture-overview.md index 99fdc8de886d2..7dd93f7115eee 100644 --- a/site2/website/versioned_docs/version-2.6.0/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.6.0/concepts-architecture-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Architecture" original_id: concepts-architecture-overview --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -17,20 +17,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -40,9 +40,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -130,17 +130,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -148,7 +148,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.6.0/concepts-authentication.md b/site2/website/versioned_docs/version-2.6.0/concepts-authentication.md index 5b94e89e16934..d48e25a1dff9d 100644 --- a/site2/website/versioned_docs/version-2.6.0/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.6.0/concepts-authentication.md @@ -5,5 +5,5 @@ sidebar_label: "Authentication and Authorization" original_id: concepts-authentication --- -Pulsar supports a pluggable [authentication](security-overview) mechanism which can be configured at broker and it also supports authorization to identify client and its access rights on topics and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at broker and it also supports authorization to identify client and its access rights on topics and tenants. diff --git a/site2/website/versioned_docs/version-2.6.0/concepts-clients.md b/site2/website/versioned_docs/version-2.6.0/concepts-clients.md index 56faae16cce44..ba287393f8e24 100644 --- a/site2/website/versioned_docs/version-2.6.0/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.6.0/concepts-clients.md @@ -5,7 +5,7 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. @@ -38,7 +38,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and will not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This will cause readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with have a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and will not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This will cause readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with have a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is just to allow users to know how behind the reader is and is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.6.0/concepts-messaging.md b/site2/website/versioned_docs/version-2.6.0/concepts-messaging.md index 217f06d5b2105..cb8c05ea4ba14 100644 --- a/site2/website/versioned_docs/version-2.6.0/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.6.0/concepts-messaging.md @@ -15,8 +15,8 @@ Messages are the basic "unit" of Pulsar. Messages are what producers publish to Component | Purpose :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages carry raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | Messages can optionally be tagged with keys, which can be useful for things like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages carry raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | Messages can optionally be tagged with keys, which can be useful for things like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer that produced the message (producers are automatically given default names, but you can apply your own explicitly as well). Sequence ID | Each Pulsar message belongs to an ordered sequence on its topic. A message's sequence ID is its ordering in that sequence. @@ -24,7 +24,7 @@ Publish time | The timestamp of when the message was published (automatically ap Event time | An optional timestamp that applications can attach to the message representing when something happened, for example, when the message was processed. The event time of a message is 0 if none is explicitly set. TypedMessageBuilder | `TypedMessageBuilder` is used to construct a message. You can set message properties like the message key, message value with `TypedMessageBuilder`.
    When you set `TypedMessageBuilder`, the best practice is to set the key as a string. If you set the key as other types, for example, an AVRO object, the key is sent as bytes, and it is difficult to get the AVRO object back on the consumer. -> For a more in-depth breakdown of Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol). +> For a more in-depth breakdown of Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -77,7 +77,7 @@ Messages can be received from [brokers](reference-terminology.md#broker) either ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -359,7 +359,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -371,7 +371,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -409,7 +409,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -464,7 +464,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -487,12 +487,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, which means that you don't need to modify your Pulsar client code. Instead, you only need to make administrative changes (see the [Managing message deduplication](cookbooks-deduplication) cookbook for a guide). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, which means that you don't need to modify your Pulsar client code. Instead, you only need to make administrative changes (see the [Managing message deduplication](cookbooks-deduplication.md) cookbook for a guide). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.6.0/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.6.0/concepts-multi-tenancy.md index 685c3031d4aad..8cba09547d2d3 100644 --- a/site2/website/versioned_docs/version-2.6.0/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.6.0/concepts-multi-tenancy.md @@ -5,7 +5,7 @@ sidebar_label: "Multi Tenancy" original_id: concepts-multi-tenancy --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -21,7 +21,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -29,7 +29,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.6.0/concepts-overview.md b/site2/website/versioned_docs/version-2.6.0/concepts-overview.md index c1262b38daf16..e8a2f4b9d321a 100644 --- a/site2/website/versioned_docs/version-2.6.0/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.6.0/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.6.0/concepts-replication.md b/site2/website/versioned_docs/version-2.6.0/concepts-replication.md index 6e23962e99771..799f0eb4d92c6 100644 --- a/site2/website/versioned_docs/version-2.6.0/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.6.0/concepts-replication.md @@ -5,5 +5,5 @@ sidebar_label: "Geo Replication" original_id: concepts-replication --- -Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that. +Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo.md) in Pulsar enables you to do that. diff --git a/site2/website/versioned_docs/version-2.6.0/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.6.0/concepts-tiered-storage.md index 0b45b0a2ed501..f6988e53a8cd4 100644 --- a/site2/website/versioned_docs/version-2.6.0/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.6.0/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.6.0/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.6.0/concepts-topic-compaction.md index e402c965384fd..34b7ed7fbbd31 100644 --- a/site2/website/versioned_docs/version-2.6.0/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.6.0/concepts-topic-compaction.md @@ -7,7 +7,7 @@ original_id: concepts-topic-compaction Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -24,7 +24,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.6.0/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.6.0/cookbooks-compaction.md index f436568e393de..864fb75255bfa 100644 --- a/site2/website/versioned_docs/version-2.6.0/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.6.0/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.6.0/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.6.0/cookbooks-non-persistent.md index 8eb1307c3eae5..f831119681634 100644 --- a/site2/website/versioned_docs/version-2.6.0/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.6.0/cookbooks-non-persistent.md @@ -41,7 +41,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-non-persistent-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-non-persistent-topics.md) guide. ## Enabling @@ -55,7 +55,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.6.0/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.6.0/cookbooks-partitioned.md index 4649d84057c84..cdf3bf3d14005 100644 --- a/site2/website/versioned_docs/version-2.6.0/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.6.0/cookbooks-partitioned.md @@ -7,7 +7,7 @@ original_id: cookbooks-partitioned By default, Pulsar topics are served by a single broker. Using only a single broker limits a topic's maximum throughput. *Partitioned topics* are a special type of topic that can span multiple brokers and thus allow for much higher throughput. For an explanation of how partitioned topics work, see the [Partitioned Topics](concepts-messaging.md#partitioned-topics) concepts. -You can publish to partitioned topics using Pulsar client libraries and you can [create and manage](#managing-partitioned-topics) partitioned topics using Pulsar [admin API](admin-api-overview). +You can publish to partitioned topics using Pulsar client libraries and you can [create and manage](#managing-partitioned-topics) partitioned topics using Pulsar [admin API](admin-api-overview.md). ## Publish to partitioned topics @@ -100,4 +100,4 @@ If a message has a key, it supersedes the round robin routing policy. The follow ## Manage partitioned topics -You can use Pulsar [admin API](admin-api-overview.md) to create and manage [partitioned topics](admin-api-partitioned-topics). \ No newline at end of file +You can use Pulsar [admin API](admin-api-overview.md) to create and manage [partitioned topics](admin-api-partitioned-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.6.0/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.6.0/cookbooks-retention-expiry.md index 791ee1020147f..03ae4771ab13b 100644 --- a/site2/website/versioned_docs/version-2.6.0/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.6.0/cookbooks-retention-expiry.md @@ -16,7 +16,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.6.0/deploy-aws.md b/site2/website/versioned_docs/version-2.6.0/deploy-aws.md index 542f40b1187ea..0ac377375f585 100644 --- a/site2/website/versioned_docs/version-2.6.0/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.6.0/deploy-aws.md @@ -5,7 +5,7 @@ sidebar_label: "Amazon Web Services" original_id: deploy-aws --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. diff --git a/site2/website/versioned_docs/version-2.6.0/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.6.0/deploy-bare-metal-multi-cluster.md index bdf8dfa251b3c..8d110c5d935bb 100644 --- a/site2/website/versioned_docs/version-2.6.0/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.6.0/deploy-bare-metal-multi-cluster.md @@ -9,17 +9,17 @@ original_id: deploy-bare-metal-multi-cluster 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using it in a startup or on a single team, you had better opt for a single cluster. For instructions on deploying a single cluster, -see the guide [here](deploy-bare-metal). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo). Deploying a multi-cluster Pulsar instance involves the following basic steps: +A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md). Deploying a multi-cluster Pulsar instance involves the following basic steps: * Deploying two separate [ZooKeeper](#deploy-zookeeper) quorums: a [local](#deploy-local-zookeeper) quorum for each cluster in the instance and a [configuration store](#configuration-store) quorum for instance-wide tasks * Initializing [cluster metadata](#cluster-metadata-initialization) for each cluster @@ -29,7 +29,7 @@ A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You If you want to deploy a single Pulsar cluster, see [Clusters and Brokers](getting-started-standalone.md#start-the-cluster). > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). ## System requirement Currently, Pulsar is available for 64-bit **macOS**, **Linux**, and **Windows**. To use Pulsar, you need to install 64-bit JRE/JDK 8 or later versions. @@ -67,7 +67,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -245,7 +245,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -364,17 +364,17 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). -You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > #### Service discovery already provided by many scheduling systems -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ### Service discovery setup -The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol) of Pulsar. +The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol.md) of Pulsar. To get started setting up the built-in service of discovery of Pulsar, you need to change a few parameters in the [`conf/discovery.conf`](reference-configuration.md#service-discovery) configuration file. Set the [`zookeeperServers`](reference-configuration.md#service-discovery-zookeeperServers) parameter to the ZooKeeper quorum connection string of the cluster and the [`configurationStoreServers`](reference-configuration.md#service-discovery-configurationStoreServers) setting to the [configuration store](reference-terminology.md#configuration-store) quorum connection string. @@ -399,7 +399,7 @@ $ bin/pulsar-daemon start discovery ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.6.0/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.6.0/deploy-bare-metal.md index 35447d6e3e5de..a903c5b5fe0d2 100644 --- a/site2/website/versioned_docs/version-2.6.0/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.6.0/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using Pulsar in a startup or on a single team, it is simplest to opt for a single cluster. If you do need to run a multi-cluster Pulsar instance, -see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -101,7 +101,7 @@ The extracted directory contains the following subdirectories: Directory | Contains :---------|:-------- -`bin` |[command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` |[command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `data` | The data storage directory that ZooKeeper and BookKeeper use `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -263,9 +263,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (you had better not use a different port). -`--web-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). +`--web-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should not use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (you had better not use a different port). -`--broker-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). +`--broker-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). > If you do not have a DNS server, you can use multi-host format in the service URL with the following settings: @@ -389,7 +389,7 @@ webServicePortTls=8443 ### Enable Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -407,7 +407,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker). +If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker.md). ### Start Brokers diff --git a/site2/website/versioned_docs/version-2.6.0/deploy-dcos.md b/site2/website/versioned_docs/version-2.6.0/deploy-dcos.md index f5f8d1faa7dbe..952d5f47e30fa 100644 --- a/site2/website/versioned_docs/version-2.6.0/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.6.0/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -If you want to enable all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of +If you want to enable all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.6.0/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.6.0/deploy-kubernetes.md index dc7123d000467..1aefc6ad79f71 100644 --- a/site2/website/versioned_docs/version-2.6.0/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.6.0/deploy-kubernetes.md @@ -6,6 +6,6 @@ original_id: deploy-kubernetes --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.6.0/deploy-monitoring.md b/site2/website/versioned_docs/version-2.6.0/deploy-monitoring.md index 2efc328097c00..40e918d9434fd 100644 --- a/site2/website/versioned_docs/version-2.6.0/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.6.0/deploy-monitoring.md @@ -73,7 +73,7 @@ The default port for bookie is `8000` (instead of `8080`). You can change the po You can use Prometheus to collect and store the metrics data. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes that needs to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is automatically setup with the [provided](deploy-kubernetes) instructions. +When you run Pulsar on bare metal, you can provide the list of nodes that needs to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is automatically setup with the [provided](deploy-kubernetes.md) instructions. ## Dashboards @@ -83,7 +83,7 @@ For that reason you only need to collect time series of metrics aggregated at th ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Dashboard](administration-dashboard). +The per-topic dashboard instructions are available at [Dashboard](administration-dashboard.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.6.0/develop-schema.md b/site2/website/versioned_docs/version-2.6.0/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.6.0/develop-schema.md +++ b/site2/website/versioned_docs/version-2.6.0/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.6.0/functions-deploy.md b/site2/website/versioned_docs/version-2.6.0/functions-deploy.md index 51f1140693dbe..d9496b3ed5b48 100644 --- a/site2/website/versioned_docs/version-2.6.0/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.6.0/functions-deploy.md @@ -9,12 +9,12 @@ original_id: functions-deploy To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface diff --git a/site2/website/versioned_docs/version-2.6.0/functions-develop.md b/site2/website/versioned_docs/version-2.6.0/functions-develop.md index 00d1abad193ad..468d91772dafa 100644 --- a/site2/website/versioned_docs/version-2.6.0/functions-develop.md +++ b/site2/website/versioned_docs/version-2.6.0/functions-develop.md @@ -856,7 +856,7 @@ public class MetricRecorderFunction implements Function { ``` -> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring) guide. +> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring.md) guide.     @@ -887,11 +887,11 @@ Currently, the feature is not available in Go. ```` ### Access metrics -To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring) in Pulsar. +To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring.md) in Pulsar. ## Security -If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker). For more details, refer to [Security settings](functions-worker.md#security-settings). +If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings). Pulsar Functions can support the following providers: diff --git a/site2/website/versioned_docs/version-2.6.0/functions-overview.md b/site2/website/versioned_docs/version-2.6.0/functions-overview.md index 25dc6029acf70..816d301e0fd0e 100644 --- a/site2/website/versioned_docs/version-2.6.0/functions-overview.md +++ b/site2/website/versioned_docs/version-2.6.0/functions-overview.md @@ -164,7 +164,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.6.0/functions-worker.md b/site2/website/versioned_docs/version-2.6.0/functions-worker.md index 0aa9fcc585135..931cdb6542d8f 100644 --- a/site2/website/versioned_docs/version-2.6.0/functions-worker.md +++ b/site2/website/versioned_docs/version-2.6.0/functions-worker.md @@ -4,7 +4,7 @@ title: Deploy and manage functions worker sidebar_label: "Setup: Pulsar Functions Worker" original_id: functions-worker --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either of the two options based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -168,7 +168,7 @@ tlsTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -195,7 +195,7 @@ properties: ``` For *Token Authentication* prodivder, add necessary settings under `properties` if needed. -See [Token Authentication](security-jwt) for more details. +See [Token Authentication](security-jwt.md) for more details. ``` diff --git a/site2/website/versioned_docs/version-2.6.0/getting-started-docker.md b/site2/website/versioned_docs/version-2.6.0/getting-started-docker.md index b9f12f0b52688..4f20971d75330 100644 --- a/site2/website/versioned_docs/version-2.6.0/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.6.0/getting-started-docker.md @@ -50,8 +50,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -110,7 +110,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.6.0/getting-started-helm.md b/site2/website/versioned_docs/version-2.6.0/getting-started-helm.md index be450cab1782c..8e59dacdfee2a 100644 --- a/site2/website/versioned_docs/version-2.6.0/getting-started-helm.md +++ b/site2/website/versioned_docs/version-2.6.0/getting-started-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -378,7 +378,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.6.0/getting-started-standalone.md b/site2/website/versioned_docs/version-2.6.0/getting-started-standalone.md index dad19d422f359..0921bb374b98b 100644 --- a/site2/website/versioned_docs/version-2.6.0/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.6.0/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -59,7 +59,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.6.0/helm-install.md b/site2/website/versioned_docs/version-2.6.0/helm-install.md index edb12260f010e..1f4d5eb69d5dd 100644 --- a/site2/website/versioned_docs/version-2.6.0/helm-install.md +++ b/site2/website/versioned_docs/version-2.6.0/helm-install.md @@ -21,7 +21,7 @@ Before deploying Pulsar, you need to prepare your environment. ### Tools -Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools) on your computer. +Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools.md) on your computer. ## Cloud cluster preparation @@ -37,8 +37,8 @@ To create and connect to the Kubernetes cluster, follow the instructions: ## Pulsar deployment -Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy). +Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy.md). ## Pulsar upgrade -To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade). +To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade.md). diff --git a/site2/website/versioned_docs/version-2.6.0/helm-overview.md b/site2/website/versioned_docs/version-2.6.0/helm-overview.md index 53633e28825f2..a07846a182b18 100644 --- a/site2/website/versioned_docs/version-2.6.0/helm-overview.md +++ b/site2/website/versioned_docs/version-2.6.0/helm-overview.md @@ -53,9 +53,9 @@ It includes support for: ## Pulsar Helm chart quick start -To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install). +This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install.md). ## Troubleshooting @@ -65,7 +65,7 @@ We have done our best to make these charts as seamless as possible. Occasionally The Apache Pulsar Helm chart contains all required dependencies. -If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm) for your first iteration. +If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm.md) for your first iteration. 1. [Preparation](helm-prepare.md) 2. [Deployment](helm-deploy.md) @@ -83,7 +83,7 @@ helm upgrade apache/pulsar -f pulsar.yaml ``` -For more detailed information, see [Upgrading](helm-upgrade). +For more detailed information, see [Upgrading](helm-upgrade.md). ## Uninstallation diff --git a/site2/website/versioned_docs/version-2.6.0/helm-prepare.md b/site2/website/versioned_docs/version-2.6.0/helm-prepare.md index 95d7e668f9eca..5e9f2f9ef4f68 100644 --- a/site2/website/versioned_docs/version-2.6.0/helm-prepare.md +++ b/site2/website/versioned_docs/version-2.6.0/helm-prepare.md @@ -89,4 +89,4 @@ PROJECT= USE_LOCAL_SSD=true LOCAL_SSD_COUNT= ## Next Steps -Continue with the [installation of the chart](helm-deploy) once you have the cluster up and running. +Continue with the [installation of the chart](helm-deploy.md) once you have the cluster up and running. diff --git a/site2/website/versioned_docs/version-2.6.0/helm-tools.md b/site2/website/versioned_docs/version-2.6.0/helm-tools.md index efd61eae32506..6ba89006913b6 100644 --- a/site2/website/versioned_docs/version-2.6.0/helm-tools.md +++ b/site2/website/versioned_docs/version-2.6.0/helm-tools.md @@ -25,7 +25,7 @@ You can get Helm from the project's [releases page](https://github.com/helm/helm ### Next steps -Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare). +Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare.md). ## Additional information diff --git a/site2/website/versioned_docs/version-2.6.0/helm-upgrade.md b/site2/website/versioned_docs/version-2.6.0/helm-upgrade.md index e39cf97407355..7d671e6bfb3c1 100644 --- a/site2/website/versioned_docs/version-2.6.0/helm-upgrade.md +++ b/site2/website/versioned_docs/version-2.6.0/helm-upgrade.md @@ -20,7 +20,7 @@ You can retrieve your previous `--set` arguments cleanly, with `helm get values To upgrade Apache Pulsar to a newer version, follow these steps: 1. Check the change log for the specific version you would like to upgrade to. -2. Go through [deployment documentation](helm-deploy) step by step. +2. Go through [deployment documentation](helm-deploy.md) step by step. 3. Extract your previous `--set` arguments with the following command. ```bash diff --git a/site2/website/versioned_docs/version-2.6.0/io-cdc.md b/site2/website/versioned_docs/version-2.6.0/io-cdc.md index 20f16ae0d6904..e6e662884826d 100644 --- a/site2/website/versioned_docs/version-2.6.0/io-cdc.md +++ b/site2/website/versioned_docs/version-2.6.0/io-cdc.md @@ -13,8 +13,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.6.0/io-develop.md b/site2/website/versioned_docs/version-2.6.0/io-develop.md index 4a815b3b9d6f6..7a356662560c9 100644 --- a/site2/website/versioned_docs/version-2.6.0/io-develop.md +++ b/site2/website/versioned_docs/version-2.6.0/io-develop.md @@ -8,15 +8,15 @@ original_id: io-develop This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -163,7 +163,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -192,7 +192,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). diff --git a/site2/website/versioned_docs/version-2.6.0/io-overview.md b/site2/website/versioned_docs/version-2.6.0/io-overview.md index 68df2472d6334..b538604edc46f 100644 --- a/site2/website/versioned_docs/version-2.6.0/io-overview.md +++ b/site2/website/versioned_docs/version-2.6.0/io-overview.md @@ -158,7 +158,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.6.0/io-quickstart.md b/site2/website/versioned_docs/version-2.6.0/io-quickstart.md index 6f976ce2c86a8..e81132d0f674a 100644 --- a/site2/website/versioned_docs/version-2.6.0/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.6.0/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you are able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you are able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However, all the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -109,7 +109,7 @@ This section demonstrates how to connect Pulsar to Cassandra. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink.md). ::: @@ -236,11 +236,11 @@ You can create a configuration file through one of the following methods. ``` -For more information, see [Cassandra sink connector](io-cassandra-sink). +For more information, see [Cassandra sink connector](io-cassandra-sink.md). ### Create a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on them. Run the following command to create a Cassandra sink connector with sink type _cassandra_ and the config file _examples/cassandra-sink.yml_ created previously. @@ -268,7 +268,7 @@ as a Pulsar Function and writes the messages produced in the topic _test_cassand ### Inspect a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * Get the information of a Cassandra sink. @@ -432,7 +432,7 @@ to monitor a connector and perform other operations on it. ### Delete a Cassandra Sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. ```bash @@ -455,7 +455,7 @@ This section demonstrates how to connect Pulsar to PostgreSQL. ::: ->For more information, see [JDBC sink connector](io-jdbc-sink). +>For more information, see [JDBC sink connector](io-jdbc-sink.md). ### Setup a PostgreSQL cluster @@ -615,7 +615,7 @@ In this section, you need to configure a JDBC sink connector. ### Create a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on it. This example creates a sink connector and specifies the desired information. @@ -661,7 +661,7 @@ The sink has been created successfully if the following message appears. ### Inspect a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * List all running JDBC sink(s). @@ -781,7 +781,7 @@ to monitor a connector and perform other operations on it. ### Stop a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to stop a connector and perform other operations on it. ```bash @@ -809,7 +809,7 @@ The sink instance has been stopped successfully if the following message disappe ### Restart a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to restart a connector and perform other operations on it. ```bash @@ -845,7 +845,7 @@ Note that `pulsar-admin sinks localrun options` **runs a sink connector locally* ### Update a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to update a connector and perform other operations on it. This example updates the parallelism of the _pulsar-postgres-jdbc-sink_ sink connector to 2. @@ -913,7 +913,7 @@ The result shows that the parallelism is 2. ### Delete a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. This example deletes the _pulsar-postgres-jdbc-sink_ sink connector. diff --git a/site2/website/versioned_docs/version-2.6.0/io-use.md b/site2/website/versioned_docs/version-2.6.0/io-use.md index 4f1a040bf991f..cae89c6723ea3 100644 --- a/site2/website/versioned_docs/version-2.6.0/io-use.md +++ b/site2/website/versioned_docs/version-2.6.0/io-use.md @@ -15,7 +15,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.6.0/kubernetes-helm.md b/site2/website/versioned_docs/version-2.6.0/kubernetes-helm.md index a9f768e6a96b3..ea92a0968cd7d 100644 --- a/site2/website/versioned_docs/version-2.6.0/kubernetes-helm.md +++ b/site2/website/versioned_docs/version-2.6.0/kubernetes-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.6.0/pulsar-2.0.md b/site2/website/versioned_docs/version-2.6.0/pulsar-2.0.md index 11c5e66cb62c2..752590f57b558 100644 --- a/site2/website/versioned_docs/version-2.6.0/pulsar-2.0.md +++ b/site2/website/versioned_docs/version-2.6.0/pulsar-2.0.md @@ -5,13 +5,13 @@ sidebar_label: "Pulsar 2.0" original_id: pulsar-2.0 --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.6.0/reference-cli-tools.md b/site2/website/versioned_docs/version-2.6.0/reference-cli-tools.md index 7541f34be0e3d..903502936de0e 100644 --- a/site2/website/versioned_docs/version-2.6.0/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.6.0/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.6.0/schema-get-started.md b/site2/website/versioned_docs/version-2.6.0/schema-get-started.md index 97e270586a26d..afacb0fa51f2e 100644 --- a/site2/website/versioned_docs/version-2.6.0/schema-get-started.md +++ b/site2/website/versioned_docs/version-2.6.0/schema-get-started.md @@ -17,7 +17,7 @@ Applications typically adopt one of the following approaches to guarantee type s #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ### Client-side approach diff --git a/site2/website/versioned_docs/version-2.6.0/schema-manage.md b/site2/website/versioned_docs/version-2.6.0/schema-manage.md index 662b8b523726d..84d0ccdc23164 100644 --- a/site2/website/versioned_docs/version-2.6.0/schema-manage.md +++ b/site2/website/versioned_docs/version-2.6.0/schema-manage.md @@ -160,7 +160,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.6.0/schema-understand.md b/site2/website/versioned_docs/version-2.6.0/schema-understand.md index fda3d6dfecb4c..2b5fd1f44b360 100644 --- a/site2/website/versioned_docs/version-2.6.0/schema-understand.md +++ b/site2/website/versioned_docs/version-2.6.0/schema-understand.md @@ -372,7 +372,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```java diff --git a/site2/website/versioned_docs/version-2.6.0/security-athenz.md b/site2/website/versioned_docs/version-2.6.0/security-athenz.md index 947c3f470be46..8a39fe25316d0 100644 --- a/site2/website/versioned_docs/version-2.6.0/security-athenz.md +++ b/site2/website/versioned_docs/version-2.6.0/security-athenz.md @@ -76,7 +76,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.6.0/security-authorization.md b/site2/website/versioned_docs/version-2.6.0/security-authorization.md index 28200ad0744b9..ef78adc6cad0c 100644 --- a/site2/website/versioned_docs/version-2.6.0/security-authorization.md +++ b/site2/website/versioned_docs/version-2.6.0/security-authorization.md @@ -27,7 +27,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you can not only use superuser roles for administrators and clients but also for broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you can not only use superuser roles for administrators and clients but also for broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. The broker still checks the authorization of the request when the broker receives the forwarded request. @@ -58,7 +58,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -86,7 +86,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication diff --git a/site2/website/versioned_docs/version-2.6.0/security-bouncy-castle.md b/site2/website/versioned_docs/version-2.6.0/security-bouncy-castle.md index 1012f0e4933b6..eaf4f907f17e4 100644 --- a/site2/website/versioned_docs/version-2.6.0/security-bouncy-castle.md +++ b/site2/website/versioned_docs/version-2.6.0/security-bouncy-castle.md @@ -16,7 +16,7 @@ In Pulsar, security and crypto have dependencies on BouncyCastle Jars. For the d `Bouncy Castle` provides both [FIPS](https://www.bouncycastle.org/fips_faq.html) and non-FIPS version. But in a JVM, you can not include both of the 2 versions, and you need to exclude the current version before include the other. -In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. +In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption.md). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. ## Include dependencies of BC-non-FIPS diff --git a/site2/website/versioned_docs/version-2.6.0/security-extending.md b/site2/website/versioned_docs/version-2.6.0/security-extending.md index 57b4c6ad928ee..37930bb286576 100644 --- a/site2/website/versioned_docs/version-2.6.0/security-extending.md +++ b/site2/website/versioned_docs/version-2.6.0/security-extending.md @@ -9,7 +9,7 @@ Pulsar provides a way to use custom authentication and authorization mechanisms. ## Authentication -Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview). +Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview.md). You can choose to use a custom authentication mechanism by providing the implementation in the form of two plugins. One plugin is for the Client library and the other plugin is for the Pulsar Broker to validate the credentials. diff --git a/site2/website/versioned_docs/version-2.6.0/security-jwt.md b/site2/website/versioned_docs/version-2.6.0/security-jwt.md index 546e72449165a..17ea2cbf6228c 100644 --- a/site2/website/versioned_docs/version-2.6.0/security-jwt.md +++ b/site2/website/versioned_docs/version-2.6.0/security-jwt.md @@ -32,11 +32,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -303,7 +303,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.6.0/security-kerberos.md b/site2/website/versioned_docs/version-2.6.0/security-kerberos.md index 670586a9d1691..c49fa3bea1fce 100644 --- a/site2/website/versioned_docs/version-2.6.0/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.6.0/security-kerberos.md @@ -373,7 +373,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.6.0/security-overview.md b/site2/website/versioned_docs/version-2.6.0/security-overview.md index 964f579305b28..91dc68e51d9cf 100644 --- a/site2/website/versioned_docs/version-2.6.0/security-overview.md +++ b/site2/website/versioned_docs/version-2.6.0/security-overview.md @@ -17,7 +17,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.6.0/security-tls-authentication.md b/site2/website/versioned_docs/version-2.6.0/security-tls-authentication.md index 20a6604a6c1fe..e1b67ffa05b78 100644 --- a/site2/website/versioned_docs/version-2.6.0/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.6.0/security-tls-authentication.md @@ -7,9 +7,9 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -101,7 +101,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -121,7 +121,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.6.0/security-tls-keystore.md b/site2/website/versioned_docs/version-2.6.0/security-tls-keystore.md index e6412ab6a86a7..180348ca708b8 100644 --- a/site2/website/versioned_docs/version-2.6.0/security-tls-keystore.md +++ b/site2/website/versioned_docs/version-2.6.0/security-tls-keystore.md @@ -7,7 +7,7 @@ original_id: security-tls-keystore ## Overview -Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication) between clients and Apache Pulsar service. +Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication.md) between clients and Apache Pulsar service. By default it uses PEM format file configuration. This page tries to describe use [KeyStore](https://en.wikipedia.org/wiki/Java_KeyStore) type configure for TLS. @@ -165,7 +165,7 @@ This is similar to [TLS encryption configuing for client with PEM type](security For a a minimal configuration, user need to provide the TrustStore information. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties @@ -258,7 +258,7 @@ webSocketServiceEnabled=false Besides the TLS encryption configuring. The main work is configuring the KeyStore, which contains a valid CN as client role, for client. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties diff --git a/site2/website/versioned_docs/version-2.6.0/security-tls-transport.md b/site2/website/versioned_docs/version-2.6.0/security-tls-transport.md index 9638b60eab1de..29fa49120c71e 100644 --- a/site2/website/versioned_docs/version-2.6.0/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.6.0/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,9 +27,9 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). -`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ## Create TLS certificates @@ -130,7 +130,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Broker Configuration -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -201,7 +201,7 @@ The examples below show hostname verification being disabled for the Java client ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.6.0/security-token-admin.md b/site2/website/versioned_docs/version-2.6.0/security-token-admin.md index 6bef23b106415..39bb6178e2bf3 100644 --- a/site2/website/versioned_docs/version-2.6.0/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.6.0/security-token-admin.md @@ -163,7 +163,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.6.0/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.6.0/sql-deployment-configurations.md index 9dafaa26507a7..0d1c7eb5f1e30 100644 --- a/site2/website/versioned_docs/version-2.6.0/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.6.0/sql-deployment-configurations.md @@ -54,7 +54,7 @@ Since Pulsar SQL is powered by [Presto](https://prestosql.io), the configuration :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.6.0/sql-overview.md b/site2/website/versioned_docs/version-2.6.0/sql-overview.md index 2bb78260265b2..8abb23ea2acbb 100644 --- a/site2/website/versioned_docs/version-2.6.0/sql-overview.md +++ b/site2/website/versioned_docs/version-2.6.0/sql-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Overview" original_id: sql-overview --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.6.0/standalone-docker.md b/site2/website/versioned_docs/version-2.6.0/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.6.0/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.6.0/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.6.0/standalone.md b/site2/website/versioned_docs/version-2.6.0/standalone.md index 220af52ce6861..e91f94140a4ac 100644 --- a/site2/website/versioned_docs/version-2.6.0/standalone.md +++ b/site2/website/versioned_docs/version-2.6.0/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -59,7 +59,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.6.1/admin-api-brokers.md b/site2/website/versioned_docs/version-2.6.1/admin-api-brokers.md index bdd6ff555ac74..9db8f153bf5da 100644 --- a/site2/website/versioned_docs/version-2.6.1/admin-api-brokers.md +++ b/site2/website/versioned_docs/version-2.6.1/admin-api-brokers.md @@ -12,7 +12,7 @@ Pulsar brokers consist of two components: [Brokers](reference-terminology.md#broker) can be managed via: -* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/brokers` endpoint of the admin {@inject: rest:REST:/} API * The `brokers` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin.html} object in the [Java API](client-libraries-java.md) @@ -95,7 +95,7 @@ One way to configure a Pulsar [broker](reference-terminology.md#broker) is to su But since all broker configuration in Pulsar is stored in ZooKeeper, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values. -* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. +* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. * In the Pulsar admin {@inject: rest:REST:/} API, dynamic configuration is managed through the `/admin/v2/brokers/configuration` endpoint. ### Update dynamic configuration diff --git a/site2/website/versioned_docs/version-2.6.1/admin-api-clusters.md b/site2/website/versioned_docs/version-2.6.1/admin-api-clusters.md index efd50a4c4b0c4..ac489a949d0f2 100644 --- a/site2/website/versioned_docs/version-2.6.1/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.6.1/admin-api-clusters.md @@ -10,7 +10,7 @@ servers (aka [bookies](reference-terminology.md#bookie)), and a [ZooKeeper](http Clusters can be managed via: -* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/clusters` endpoint of the admin {@inject: rest:REST:/} API * The `clusters` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md) @@ -62,7 +62,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > #### No cluster metadata initialization through the REST API or the Java admin API > @@ -86,11 +86,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. #### pulsar-admin diff --git a/site2/website/versioned_docs/version-2.6.1/admin-api-functions.md b/site2/website/versioned_docs/version-2.6.1/admin-api-functions.md index 96b7dc7ea52cd..315943492c214 100644 --- a/site2/website/versioned_docs/version-2.6.1/admin-api-functions.md +++ b/site2/website/versioned_docs/version-2.6.1/admin-api-functions.md @@ -15,9 +15,9 @@ Functions can be managed via the following methods. Method | Description ---|--- -**Admin CLI** | The [`functions`](reference-pulsar-admin.md#functions) command of the [`pulsar-admin`](reference-pulsar-admin) tool. +**Admin CLI** | The [`functions`](reference-pulsar-admin.md#functions) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool. **REST API** |The `/admin/v3/functions` endpoint of the admin {@inject: rest:REST:/} API. -**Java Admin API**| The `functions` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java). +**Java Admin API**| The `functions` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md). ## Function resources diff --git a/site2/website/versioned_docs/version-2.6.1/admin-api-namespaces.md b/site2/website/versioned_docs/version-2.6.1/admin-api-namespaces.md index bc8e72c0b2902..1ffad6382c2ea 100644 --- a/site2/website/versioned_docs/version-2.6.1/admin-api-namespaces.md +++ b/site2/website/versioned_docs/version-2.6.1/admin-api-namespaces.md @@ -9,7 +9,7 @@ Pulsar [namespaces](reference-terminology.md#namespace) are logical groupings of Namespaces can be managed via: -* The [`namespaces`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`namespaces`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/namespaces` endpoint of the admin {@inject: rest:REST:/} API * The `namespaces` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md) diff --git a/site2/website/versioned_docs/version-2.6.1/admin-api-non-partitioned-topics.md b/site2/website/versioned_docs/version-2.6.1/admin-api-non-partitioned-topics.md index 9cd019d2abe8f..d34d8def6473f 100644 --- a/site2/website/versioned_docs/version-2.6.1/admin-api-non-partitioned-topics.md +++ b/site2/website/versioned_docs/version-2.6.1/admin-api-non-partitioned-topics.md @@ -6,7 +6,7 @@ original_id: admin-api-non-partitioned-topics --- -You can use Pulsar's [admin API](admin-api-overview) to create and manage non-partitioned topics. +You can use Pulsar's [admin API](admin-api-overview.md) to create and manage non-partitioned topics. In all of the instructions and commands below, the topic name structure is: diff --git a/site2/website/versioned_docs/version-2.6.1/admin-api-overview.md b/site2/website/versioned_docs/version-2.6.1/admin-api-overview.md index b6186cdd0ea9a..ec19fceda0db2 100644 --- a/site2/website/versioned_docs/version-2.6.1/admin-api-overview.md +++ b/site2/website/versioned_docs/version-2.6.1/admin-api-overview.md @@ -12,7 +12,7 @@ You can currently interact with the admin interface via: - Making HTTP calls against the admin {@inject: rest:REST:/} API provided by Pulsar [brokers](reference-terminology.md#broker). For some restful apis, they might be redirected to topic owner brokers for serving with [`307 Temporary Redirect`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/307), hence the HTTP callers should handle `307 Temporary Redirect`. If you are using `curl`, you should specify `-L` to handle redirections. -- The `pulsar-admin` CLI tool, which is available in the `bin` folder of your [Pulsar installation](getting-started-standalone): +- The `pulsar-admin` CLI tool, which is available in the `bin` folder of your [Pulsar installation](getting-started-standalone.md): ```shell @@ -20,7 +20,7 @@ $ bin/pulsar-admin ``` -Full documentation for this tool can be found in the [Pulsar command-line tools](reference-pulsar-admin) doc. +Full documentation for this tool can be found in the [Pulsar command-line tools](reference-pulsar-admin.md) doc. - A Java client interface. @@ -31,11 +31,11 @@ In this document, examples from each of the three available interfaces will be s ## Admin setup -Each of Pulsar's three admin interfaces---the [`pulsar-admin`](reference-pulsar-admin) CLI tool, the [Java admin API](/api/admin), and the {@inject: rest:REST:/} API ---requires some special setup if you have [authentication](security-overview.md#authentication-providers) enabled in your Pulsar [instance](reference-terminology.md#instance). +Each of Pulsar's three admin interfaces---the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool, the [Java admin API](/api/admin), and the {@inject: rest:REST:/} API ---requires some special setup if you have [authentication](security-overview.md#authentication-providers) enabled in your Pulsar [instance](reference-terminology.md#instance). ### pulsar-admin -If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: +If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin.md) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: |Name|Description|Default| |----|-----------|-------| diff --git a/site2/website/versioned_docs/version-2.6.1/admin-api-partitioned-topics.md b/site2/website/versioned_docs/version-2.6.1/admin-api-partitioned-topics.md index eef6efdf7407b..73edab60a4039 100644 --- a/site2/website/versioned_docs/version-2.6.1/admin-api-partitioned-topics.md +++ b/site2/website/versioned_docs/version-2.6.1/admin-api-partitioned-topics.md @@ -6,7 +6,7 @@ original_id: admin-api-partitioned-topics --- -You can use Pulsar's [admin API](admin-api-overview) to create and manage partitioned topics. +You can use Pulsar's [admin API](admin-api-overview.md) to create and manage partitioned topics. In all of the instructions and commands below, the topic name structure is: diff --git a/site2/website/versioned_docs/version-2.6.1/admin-api-tenants.md b/site2/website/versioned_docs/version-2.6.1/admin-api-tenants.md index 924cbce3d41ab..27764d6da3885 100644 --- a/site2/website/versioned_docs/version-2.6.1/admin-api-tenants.md +++ b/site2/website/versioned_docs/version-2.6.1/admin-api-tenants.md @@ -5,7 +5,7 @@ sidebar_label: "Tenants" original_id: admin-api-tenants --- -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters diff --git a/site2/website/versioned_docs/version-2.6.1/administration-dashboard.md b/site2/website/versioned_docs/version-2.6.1/administration-dashboard.md index 514b076239789..25f976609b40b 100644 --- a/site2/website/versioned_docs/version-2.6.1/administration-dashboard.md +++ b/site2/website/versioned_docs/version-2.6.1/administration-dashboard.md @@ -7,7 +7,7 @@ original_id: administration-dashboard :::note -Pulsar dashboard is deprecated. If you want to manage and monitor the stats of your topics, use [Pulsar Manager](administration-pulsar-manager). +Pulsar dashboard is deprecated. If you want to manage and monitor the stats of your topics, use [Pulsar Manager](administration-pulsar-manager.md). ::: diff --git a/site2/website/versioned_docs/version-2.6.1/administration-geo.md b/site2/website/versioned_docs/version-2.6.1/administration-geo.md index 42d32304a2fa4..4170d62f3ac9c 100644 --- a/site2/website/versioned_docs/version-2.6.1/administration-geo.md +++ b/site2/website/versioned_docs/version-2.6.1/administration-geo.md @@ -135,7 +135,7 @@ Once you create a geo-replication namespace, any topics that producers or consum By default, messages are replicated to all clusters configured for the namespace. You can restrict replication selectively by specifying a replication list for a message, and then that message is replicated only to the subset in the replication list. -The following is an example for the [Java API](client-libraries-java). Note the use of the `setReplicationClusters` method when you construct the {@inject: javadoc:Message:/client/org/apache/pulsar/client/api/Message} object: +The following is an example for the [Java API](client-libraries-java.md). Note the use of the `setReplicationClusters` method when you construct the {@inject: javadoc:Message:/client/org/apache/pulsar/client/api/Message} object: ```java @@ -157,7 +157,7 @@ producer.newMessage() #### Topic stats -Topic-specific statistics for geo-replication topics are available via the [`pulsar-admin`](reference-pulsar-admin) tool and {@inject: rest:REST:/} API: +Topic-specific statistics for geo-replication topics are available via the [`pulsar-admin`](reference-pulsar-admin.md) tool and {@inject: rest:REST:/} API: ```shell diff --git a/site2/website/versioned_docs/version-2.6.1/administration-zk-bk.md b/site2/website/versioned_docs/version-2.6.1/administration-zk-bk.md index 4d1038a416b9e..34cb025c7541c 100644 --- a/site2/website/versioned_docs/version-2.6.1/administration-zk-bk.md +++ b/site2/website/versioned_docs/version-2.6.1/administration-zk-bk.md @@ -174,7 +174,7 @@ The [`conf/global-zookeeper.conf`](reference-configuration.md#configuration-stor BookKeeper stores all durable messages in Pulsar. BookKeeper is a distributed [write-ahead log](https://en.wikipedia.org/wiki/Write-ahead_logging) WAL system that guarantees read consistency of independent message logs calls ledgers. Individual BookKeeper servers are also called *bookies*. -> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.6.1/client-libraries-cgo.md b/site2/website/versioned_docs/version-2.6.1/client-libraries-cgo.md index c79f7bb965473..f352f942b7714 100644 --- a/site2/website/versioned_docs/version-2.6.1/client-libraries-cgo.md +++ b/site2/website/versioned_docs/version-2.6.1/client-libraries-cgo.md @@ -24,7 +24,7 @@ Currently, the following Go clients are maintained in two repositories. ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Install go package @@ -57,7 +57,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -75,7 +75,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -236,14 +236,14 @@ Parameter | Description | Default `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | `Properties`| Attach a set of application defined properties to the producer. This properties will be visible in the topic stats | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | Set the number of max pending messages across all the partitions. This setting will be used to lower the max pending messages for each partition `MaxPendingMessages(int)`, if the total exceeds the configured value.| `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | `Batching` | Control whether automatic batching of messages is enabled for the producer. | false `BatchingMaxPublishDelay` | Set the time period within which the messages sent will be batched (default: 1ms) if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or until | 1ms `BatchingMaxMessages` | Set the maximum number of messages permitted in a batch. (default: 1000) If set to a value greater than 1, messages will be queued until this threshold is reached or batch interval has elapsed | 1000 @@ -512,7 +512,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.6.1/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.6.1/client-libraries-cpp.md index bb52d9effa404..684df5fd2d825 100644 --- a/site2/website/versioned_docs/version-2.6.1/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.6.1/client-libraries-cpp.md @@ -261,7 +261,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema -This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started). +This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started.md). ### Create producer with Avro schema diff --git a/site2/website/versioned_docs/version-2.6.1/client-libraries-dotnet.md b/site2/website/versioned_docs/version-2.6.1/client-libraries-dotnet.md index ade664cd95277..b574fa0b2e5ed 100644 --- a/site2/website/versioned_docs/version-2.6.1/client-libraries-dotnet.md +++ b/site2/website/versioned_docs/version-2.6.1/client-libraries-dotnet.md @@ -170,7 +170,7 @@ var client = PulsarClient.Builder() Currently, the Pulsar C# client supports the TLS (Transport Layer Security) and JWT (JSON Web Token) authentication. -If you have followed [Authentication using TLS](security-tls-authentication), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: +If you have followed [Authentication using TLS](security-tls-authentication.md), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: 1. Create an unencrypted and password-less pfx file. diff --git a/site2/website/versioned_docs/version-2.6.1/client-libraries-go.md b/site2/website/versioned_docs/version-2.6.1/client-libraries-go.md index 97705a643c049..6c05e1d733422 100644 --- a/site2/website/versioned_docs/version-2.6.1/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.6.1/client-libraries-go.md @@ -35,7 +35,7 @@ import "github.com/apache/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -53,7 +53,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -680,7 +680,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker @@ -700,7 +700,7 @@ opts := pulsar.ClientOptions{ ## OAuth2 authentication -To use [OAuth2 authentication](security-oauth2), you'll need to configure your client to perform the following operations. +To use [OAuth2 authentication](security-oauth2.md), you'll need to configure your client to perform the following operations. This example shows how to configure OAuth2 authentication. ```go diff --git a/site2/website/versioned_docs/version-2.6.1/client-libraries-java.md b/site2/website/versioned_docs/version-2.6.1/client-libraries-java.md index 98572b1e35b60..c13fc69fff5f0 100644 --- a/site2/website/versioned_docs/version-2.6.1/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.6.1/client-libraries-java.md @@ -5,7 +5,7 @@ sidebar_label: "Java" original_id: client-libraries-java --- -You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), and [reader](#reader) of a Java client are thread-safe. @@ -14,10 +14,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Includes both `pulsar-client` and `pulsar-client-admin`

    Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.

    In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -57,7 +57,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -83,7 +83,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -114,7 +114,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -237,7 +237,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -782,7 +782,7 @@ Total hash range size is 65536, so the max end of the range should be less than ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -896,7 +896,7 @@ The following schema formats are currently available for Java: ## Authentication -Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2). You can use the Pulsar Java client with all of them. +Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2.md). You can use the Pulsar Java client with all of them. ### TLS Authentication @@ -923,7 +923,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` @@ -960,7 +960,7 @@ PulsarClient client = PulsarClient.builder() ### Oauth2 -The following example shows how to use [Oauth2](security-oauth2) as an authentication provider for the Pulsar Java client. +The following example shows how to use [Oauth2](security-oauth2.md) as an authentication provider for the Pulsar Java client. You can use the factory method to configure authentication for Pulsar Java client. diff --git a/site2/website/versioned_docs/version-2.6.1/client-libraries-node.md b/site2/website/versioned_docs/version-2.6.1/client-libraries-node.md index 8d7376d3793e5..b5a138cb86528 100644 --- a/site2/website/versioned_docs/version-2.6.1/client-libraries-node.md +++ b/site2/website/versioned_docs/version-2.6.1/client-libraries-node.md @@ -44,7 +44,7 @@ Also, this library works only in Node.js 10.x or later because it uses the [`nod ::: ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here is an example for `localhost`: @@ -62,7 +62,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication), the URL will look like something like this: +If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication.md), the URL will look like something like this: ```http @@ -97,7 +97,7 @@ The following configurable parameters are available for Pulsar clients: | Parameter | Description | Default | | :-------- | :---------- | :------ | | `serviceUrl` | The connection URL for the Pulsar cluster. See [above](#connection-urls) for more info. | | -| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication) for more info. | | +| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication.md) for more info. | | | `operationTimeoutSeconds` | The timeout for Node.js client operations (creating producers, subscribing to and unsubscribing from [topics](reference-terminology.md#topic)). Retries will occur until this threshold is reached, at which point the operation will fail. | 30 | | `ioThreads` | The number of threads to use for handling connections to Pulsar [brokers](reference-terminology.md#broker). | 1 | | `messageListenerThreads` | The number of threads used by message listeners ([consumers](#consumers) and [readers](#readers)). | 1 | @@ -147,7 +147,7 @@ Pulsar Node.js producers have the following methods available: | :-------- | :---------- | :------ | | `topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer publishes messages. The topic format is `` or `//`. For example, `sample/ns1/my-topic`. | | | `producerName` | A name for the producer. If you do not explicitly assign a name, Pulsar will automatically generate a globally unique name. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | | -| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30000 | +| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30000 | | `initialSequenceId` | The initial sequence ID of the message. When producer send message, add sequence ID to message. The ID is increased each time to send. | | | `maxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `send` method will fail *unless* `blockIfQueueFull` is set to `true`. | 1000 | | `maxPendingMessagesAcrossPartitions` | The maximum size of the sum of partition's pending queue. | 50000 | diff --git a/site2/website/versioned_docs/version-2.6.1/client-libraries-python.md b/site2/website/versioned_docs/version-2.6.1/client-libraries-python.md index 459345ddb7f5b..6d4c7e6a77888 100644 --- a/site2/website/versioned_docs/version-2.6.1/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.6.1/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. diff --git a/site2/website/versioned_docs/version-2.6.1/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.6.1/client-libraries-websocket.md index 9cc5d4acb8ded..f939a32d19b13 100644 --- a/site2/website/versioned_docs/version-2.6.1/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.6.1/client-libraries-websocket.md @@ -5,14 +5,14 @@ sidebar_label: "WebSocket" original_id: client-libraries-websocket --- -Pulsar's [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API is meant to provide a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSockets you can publish and consume messages and use all the features available in the [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp) client libraries. +Pulsar's [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API is meant to provide a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSockets you can publish and consume messages and use all the features available in the [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md) client libraries. > You can use Pulsar's WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.6.1/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.6.1/concepts-architecture-overview.md index 99fdc8de886d2..7dd93f7115eee 100644 --- a/site2/website/versioned_docs/version-2.6.1/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.6.1/concepts-architecture-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Architecture" original_id: concepts-architecture-overview --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -17,20 +17,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -40,9 +40,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -130,17 +130,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -148,7 +148,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.6.1/concepts-authentication.md b/site2/website/versioned_docs/version-2.6.1/concepts-authentication.md index 5b94e89e16934..d48e25a1dff9d 100644 --- a/site2/website/versioned_docs/version-2.6.1/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.6.1/concepts-authentication.md @@ -5,5 +5,5 @@ sidebar_label: "Authentication and Authorization" original_id: concepts-authentication --- -Pulsar supports a pluggable [authentication](security-overview) mechanism which can be configured at broker and it also supports authorization to identify client and its access rights on topics and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at broker and it also supports authorization to identify client and its access rights on topics and tenants. diff --git a/site2/website/versioned_docs/version-2.6.1/concepts-clients.md b/site2/website/versioned_docs/version-2.6.1/concepts-clients.md index 3492c8071ca68..6149b60bb4cb4 100644 --- a/site2/website/versioned_docs/version-2.6.1/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.6.1/concepts-clients.md @@ -5,7 +5,7 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. @@ -38,7 +38,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and will not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This will cause readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with have a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and will not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This will cause readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with have a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is just to allow users to know how behind the reader is and is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.6.1/concepts-messaging.md b/site2/website/versioned_docs/version-2.6.1/concepts-messaging.md index 29cebdfbaedf8..d8104060d987a 100644 --- a/site2/website/versioned_docs/version-2.6.1/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.6.1/concepts-messaging.md @@ -15,8 +15,8 @@ Messages are the basic "unit" of Pulsar. The following table lists the component Component | Description :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer who produces the message. If you do not specify a producer name, the default name is used. Sequence ID | Each Pulsar message belongs to an ordered sequence on its topic. The sequence ID of the message is its order in that sequence. @@ -24,7 +24,7 @@ Publish time | The timestamp of when the message is published. The timestamp is Event time | An optional timestamp attached to a message by applications. For example, applications attach a timestamp on when the message is processed. If nothing is set to event time, the value is `0`. TypedMessageBuilder | It is used to construct a message. You can set message properties such as the message key, message value with `TypedMessageBuilder`.
    When you set `TypedMessageBuilder`, set the key as a string. If you set the key as other types, for example, an AVRO object, the key is sent as bytes, and it is difficult to get the AVRO object back on the consumer. -> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol). +> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -101,7 +101,7 @@ Messages are received from [brokers](reference-terminology.md#broker) either syn ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -378,7 +378,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -390,7 +390,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -428,7 +428,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -483,7 +483,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -506,12 +506,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication.md). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.6.1/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.6.1/concepts-multi-tenancy.md index 685c3031d4aad..8cba09547d2d3 100644 --- a/site2/website/versioned_docs/version-2.6.1/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.6.1/concepts-multi-tenancy.md @@ -5,7 +5,7 @@ sidebar_label: "Multi Tenancy" original_id: concepts-multi-tenancy --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -21,7 +21,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -29,7 +29,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.6.1/concepts-overview.md b/site2/website/versioned_docs/version-2.6.1/concepts-overview.md index c1262b38daf16..e8a2f4b9d321a 100644 --- a/site2/website/versioned_docs/version-2.6.1/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.6.1/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.6.1/concepts-replication.md b/site2/website/versioned_docs/version-2.6.1/concepts-replication.md index 6e23962e99771..799f0eb4d92c6 100644 --- a/site2/website/versioned_docs/version-2.6.1/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.6.1/concepts-replication.md @@ -5,5 +5,5 @@ sidebar_label: "Geo Replication" original_id: concepts-replication --- -Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that. +Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo.md) in Pulsar enables you to do that. diff --git a/site2/website/versioned_docs/version-2.6.1/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.6.1/concepts-tiered-storage.md index 0b45b0a2ed501..f6988e53a8cd4 100644 --- a/site2/website/versioned_docs/version-2.6.1/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.6.1/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.6.1/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.6.1/concepts-topic-compaction.md index e402c965384fd..34b7ed7fbbd31 100644 --- a/site2/website/versioned_docs/version-2.6.1/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.6.1/concepts-topic-compaction.md @@ -7,7 +7,7 @@ original_id: concepts-topic-compaction Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -24,7 +24,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.6.1/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.6.1/cookbooks-compaction.md index f436568e393de..864fb75255bfa 100644 --- a/site2/website/versioned_docs/version-2.6.1/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.6.1/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.6.1/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.6.1/cookbooks-non-persistent.md index 8eb1307c3eae5..f831119681634 100644 --- a/site2/website/versioned_docs/version-2.6.1/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.6.1/cookbooks-non-persistent.md @@ -41,7 +41,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-non-persistent-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-non-persistent-topics.md) guide. ## Enabling @@ -55,7 +55,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.6.1/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.6.1/cookbooks-partitioned.md index 4649d84057c84..cdf3bf3d14005 100644 --- a/site2/website/versioned_docs/version-2.6.1/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.6.1/cookbooks-partitioned.md @@ -7,7 +7,7 @@ original_id: cookbooks-partitioned By default, Pulsar topics are served by a single broker. Using only a single broker limits a topic's maximum throughput. *Partitioned topics* are a special type of topic that can span multiple brokers and thus allow for much higher throughput. For an explanation of how partitioned topics work, see the [Partitioned Topics](concepts-messaging.md#partitioned-topics) concepts. -You can publish to partitioned topics using Pulsar client libraries and you can [create and manage](#managing-partitioned-topics) partitioned topics using Pulsar [admin API](admin-api-overview). +You can publish to partitioned topics using Pulsar client libraries and you can [create and manage](#managing-partitioned-topics) partitioned topics using Pulsar [admin API](admin-api-overview.md). ## Publish to partitioned topics @@ -100,4 +100,4 @@ If a message has a key, it supersedes the round robin routing policy. The follow ## Manage partitioned topics -You can use Pulsar [admin API](admin-api-overview.md) to create and manage [partitioned topics](admin-api-partitioned-topics). \ No newline at end of file +You can use Pulsar [admin API](admin-api-overview.md) to create and manage [partitioned topics](admin-api-partitioned-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.6.1/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.6.1/cookbooks-retention-expiry.md index ae2886cca6755..5e5ff9330ed02 100644 --- a/site2/website/versioned_docs/version-2.6.1/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.6.1/cookbooks-retention-expiry.md @@ -22,7 +22,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.6.1/deploy-aws.md b/site2/website/versioned_docs/version-2.6.1/deploy-aws.md index 542f40b1187ea..0ac377375f585 100644 --- a/site2/website/versioned_docs/version-2.6.1/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.6.1/deploy-aws.md @@ -5,7 +5,7 @@ sidebar_label: "Amazon Web Services" original_id: deploy-aws --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. diff --git a/site2/website/versioned_docs/version-2.6.1/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.6.1/deploy-bare-metal-multi-cluster.md index bdf8dfa251b3c..8d110c5d935bb 100644 --- a/site2/website/versioned_docs/version-2.6.1/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.6.1/deploy-bare-metal-multi-cluster.md @@ -9,17 +9,17 @@ original_id: deploy-bare-metal-multi-cluster 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using it in a startup or on a single team, you had better opt for a single cluster. For instructions on deploying a single cluster, -see the guide [here](deploy-bare-metal). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo). Deploying a multi-cluster Pulsar instance involves the following basic steps: +A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md). Deploying a multi-cluster Pulsar instance involves the following basic steps: * Deploying two separate [ZooKeeper](#deploy-zookeeper) quorums: a [local](#deploy-local-zookeeper) quorum for each cluster in the instance and a [configuration store](#configuration-store) quorum for instance-wide tasks * Initializing [cluster metadata](#cluster-metadata-initialization) for each cluster @@ -29,7 +29,7 @@ A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You If you want to deploy a single Pulsar cluster, see [Clusters and Brokers](getting-started-standalone.md#start-the-cluster). > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). ## System requirement Currently, Pulsar is available for 64-bit **macOS**, **Linux**, and **Windows**. To use Pulsar, you need to install 64-bit JRE/JDK 8 or later versions. @@ -67,7 +67,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -245,7 +245,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -364,17 +364,17 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). -You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > #### Service discovery already provided by many scheduling systems -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ### Service discovery setup -The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol) of Pulsar. +The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol.md) of Pulsar. To get started setting up the built-in service of discovery of Pulsar, you need to change a few parameters in the [`conf/discovery.conf`](reference-configuration.md#service-discovery) configuration file. Set the [`zookeeperServers`](reference-configuration.md#service-discovery-zookeeperServers) parameter to the ZooKeeper quorum connection string of the cluster and the [`configurationStoreServers`](reference-configuration.md#service-discovery-configurationStoreServers) setting to the [configuration store](reference-terminology.md#configuration-store) quorum connection string. @@ -399,7 +399,7 @@ $ bin/pulsar-daemon start discovery ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.6.1/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.6.1/deploy-bare-metal.md index 2c341748b5605..5895cfe566bad 100644 --- a/site2/website/versioned_docs/version-2.6.1/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.6.1/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using Pulsar in a startup or on a single team, it is simplest to opt for a single cluster. If you do need to run a multi-cluster Pulsar instance, -see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -101,7 +101,7 @@ The extracted directory contains the following subdirectories: Directory | Contains :---------|:-------- -`bin` |[command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` |[command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `data` | The data storage directory that ZooKeeper and BookKeeper use `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -263,9 +263,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (you had better not use a different port). -`--web-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). +`--web-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should not use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (you had better not use a different port). -`--broker-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). +`--broker-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). > If you do not have a DNS server, you can use multi-host format in the service URL with the following settings: @@ -389,7 +389,7 @@ webServicePortTls=8443 ### Enable Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -407,7 +407,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker). +If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker.md). ### Start Brokers diff --git a/site2/website/versioned_docs/version-2.6.1/deploy-dcos.md b/site2/website/versioned_docs/version-2.6.1/deploy-dcos.md index f5f8d1faa7dbe..952d5f47e30fa 100644 --- a/site2/website/versioned_docs/version-2.6.1/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.6.1/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -If you want to enable all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of +If you want to enable all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.6.1/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.6.1/deploy-kubernetes.md index dc7123d000467..1aefc6ad79f71 100644 --- a/site2/website/versioned_docs/version-2.6.1/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.6.1/deploy-kubernetes.md @@ -6,6 +6,6 @@ original_id: deploy-kubernetes --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.6.1/deploy-monitoring.md b/site2/website/versioned_docs/version-2.6.1/deploy-monitoring.md index 2efc328097c00..40e918d9434fd 100644 --- a/site2/website/versioned_docs/version-2.6.1/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.6.1/deploy-monitoring.md @@ -73,7 +73,7 @@ The default port for bookie is `8000` (instead of `8080`). You can change the po You can use Prometheus to collect and store the metrics data. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes that needs to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is automatically setup with the [provided](deploy-kubernetes) instructions. +When you run Pulsar on bare metal, you can provide the list of nodes that needs to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is automatically setup with the [provided](deploy-kubernetes.md) instructions. ## Dashboards @@ -83,7 +83,7 @@ For that reason you only need to collect time series of metrics aggregated at th ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Dashboard](administration-dashboard). +The per-topic dashboard instructions are available at [Dashboard](administration-dashboard.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.6.1/develop-schema.md b/site2/website/versioned_docs/version-2.6.1/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.6.1/develop-schema.md +++ b/site2/website/versioned_docs/version-2.6.1/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.6.1/functions-deploy.md b/site2/website/versioned_docs/version-2.6.1/functions-deploy.md index 51f1140693dbe..d9496b3ed5b48 100644 --- a/site2/website/versioned_docs/version-2.6.1/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.6.1/functions-deploy.md @@ -9,12 +9,12 @@ original_id: functions-deploy To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface diff --git a/site2/website/versioned_docs/version-2.6.1/functions-develop.md b/site2/website/versioned_docs/version-2.6.1/functions-develop.md index dce77b52bb914..ca885cfdc2b04 100644 --- a/site2/website/versioned_docs/version-2.6.1/functions-develop.md +++ b/site2/website/versioned_docs/version-2.6.1/functions-develop.md @@ -852,7 +852,7 @@ public class MetricRecorderFunction implements Function { ``` -> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring) guide. +> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring.md) guide.     @@ -883,11 +883,11 @@ Currently, the feature is not available in Go. ```` ### Access metrics -To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring) in Pulsar. +To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring.md) in Pulsar. ## Security -If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker). For more details, refer to [Security settings](functions-worker.md#security-settings). +If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings). Pulsar Functions can support the following providers: diff --git a/site2/website/versioned_docs/version-2.6.1/functions-overview.md b/site2/website/versioned_docs/version-2.6.1/functions-overview.md index 25dc6029acf70..816d301e0fd0e 100644 --- a/site2/website/versioned_docs/version-2.6.1/functions-overview.md +++ b/site2/website/versioned_docs/version-2.6.1/functions-overview.md @@ -164,7 +164,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.6.1/functions-worker.md b/site2/website/versioned_docs/version-2.6.1/functions-worker.md index e6a448955b4bf..2b8968a467648 100644 --- a/site2/website/versioned_docs/version-2.6.1/functions-worker.md +++ b/site2/website/versioned_docs/version-2.6.1/functions-worker.md @@ -4,7 +4,7 @@ title: Deploy and manage functions worker sidebar_label: "Setup: Pulsar Functions Worker" original_id: functions-worker --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -178,7 +178,7 @@ brokerClientTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -198,7 +198,7 @@ authenticationProviders: [ provider1, provider2 ] ``` For *TLS Authentication* provider, follow the example below to add the necessary settings. -See [TLS Authentication](security-tls-authentication) for more details. +See [TLS Authentication](security-tls-authentication.md) for more details. ``` @@ -222,7 +222,7 @@ properties: ``` For *Token Authentication* provider, add necessary settings for `properties` if needed. -See [Token Authentication](security-jwt) for more details. +See [Token Authentication](security-jwt.md) for more details. ``` diff --git a/site2/website/versioned_docs/version-2.6.1/getting-started-docker.md b/site2/website/versioned_docs/version-2.6.1/getting-started-docker.md index b9f12f0b52688..4f20971d75330 100644 --- a/site2/website/versioned_docs/version-2.6.1/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.6.1/getting-started-docker.md @@ -50,8 +50,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -110,7 +110,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.6.1/getting-started-helm.md b/site2/website/versioned_docs/version-2.6.1/getting-started-helm.md index cb4c6945f2022..d03266c1e814e 100644 --- a/site2/website/versioned_docs/version-2.6.1/getting-started-helm.md +++ b/site2/website/versioned_docs/version-2.6.1/getting-started-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -383,7 +383,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.6.1/getting-started-standalone.md b/site2/website/versioned_docs/version-2.6.1/getting-started-standalone.md index dad19d422f359..0921bb374b98b 100644 --- a/site2/website/versioned_docs/version-2.6.1/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.6.1/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -59,7 +59,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.6.1/helm-install.md b/site2/website/versioned_docs/version-2.6.1/helm-install.md index cb72f140967e9..4559b145d09a5 100644 --- a/site2/website/versioned_docs/version-2.6.1/helm-install.md +++ b/site2/website/versioned_docs/version-2.6.1/helm-install.md @@ -21,7 +21,7 @@ Before deploying Pulsar, you need to prepare your environment. ### Tools -Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools) on your computer. +Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools.md) on your computer. ## Cloud cluster preparation @@ -37,8 +37,8 @@ To create and connect to the Kubernetes cluster, follow the instructions: ## Pulsar deployment -Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy). +Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy.md). ## Pulsar upgrade -To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade). +To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade.md). diff --git a/site2/website/versioned_docs/version-2.6.1/helm-overview.md b/site2/website/versioned_docs/version-2.6.1/helm-overview.md index a77499e81ae7a..eb7b8d2ca91c7 100644 --- a/site2/website/versioned_docs/version-2.6.1/helm-overview.md +++ b/site2/website/versioned_docs/version-2.6.1/helm-overview.md @@ -53,9 +53,9 @@ It includes support for: ## Pulsar Helm chart quick start -To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install). +This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install.md). ## Troubleshooting @@ -65,7 +65,7 @@ We have done our best to make these charts as seamless as possible. Occasionally The Apache Pulsar Helm chart contains all required dependencies. -If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm) for your first iteration. +If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm.md) for your first iteration. 1. [Preparation](helm-prepare.md) 2. [Deployment](helm-deploy.md) @@ -83,7 +83,7 @@ helm upgrade pulsar -f pulsar.yaml ``` -For more detailed information, see [Upgrading](helm-upgrade). +For more detailed information, see [Upgrading](helm-upgrade.md). ## Uninstallation diff --git a/site2/website/versioned_docs/version-2.6.1/helm-prepare.md b/site2/website/versioned_docs/version-2.6.1/helm-prepare.md index 95d7e668f9eca..5e9f2f9ef4f68 100644 --- a/site2/website/versioned_docs/version-2.6.1/helm-prepare.md +++ b/site2/website/versioned_docs/version-2.6.1/helm-prepare.md @@ -89,4 +89,4 @@ PROJECT= USE_LOCAL_SSD=true LOCAL_SSD_COUNT= ## Next Steps -Continue with the [installation of the chart](helm-deploy) once you have the cluster up and running. +Continue with the [installation of the chart](helm-deploy.md) once you have the cluster up and running. diff --git a/site2/website/versioned_docs/version-2.6.1/helm-tools.md b/site2/website/versioned_docs/version-2.6.1/helm-tools.md index efd61eae32506..6ba89006913b6 100644 --- a/site2/website/versioned_docs/version-2.6.1/helm-tools.md +++ b/site2/website/versioned_docs/version-2.6.1/helm-tools.md @@ -25,7 +25,7 @@ You can get Helm from the project's [releases page](https://github.com/helm/helm ### Next steps -Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare). +Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare.md). ## Additional information diff --git a/site2/website/versioned_docs/version-2.6.1/helm-upgrade.md b/site2/website/versioned_docs/version-2.6.1/helm-upgrade.md index 3ae1cdf2ba2fd..18838f363fb5f 100644 --- a/site2/website/versioned_docs/version-2.6.1/helm-upgrade.md +++ b/site2/website/versioned_docs/version-2.6.1/helm-upgrade.md @@ -20,7 +20,7 @@ You can retrieve your previous `--set` arguments cleanly, with `helm get values To upgrade Apache Pulsar to a newer version, follow these steps: 1. Check the change log for the specific version you would like to upgrade to. -2. Go through [deployment documentation](helm-deploy) step by step. +2. Go through [deployment documentation](helm-deploy.md) step by step. 3. Extract your previous `--set` arguments with the following command. ```bash diff --git a/site2/website/versioned_docs/version-2.6.1/io-cdc.md b/site2/website/versioned_docs/version-2.6.1/io-cdc.md index 20f16ae0d6904..e6e662884826d 100644 --- a/site2/website/versioned_docs/version-2.6.1/io-cdc.md +++ b/site2/website/versioned_docs/version-2.6.1/io-cdc.md @@ -13,8 +13,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.6.1/io-develop.md b/site2/website/versioned_docs/version-2.6.1/io-develop.md index 4a815b3b9d6f6..7a356662560c9 100644 --- a/site2/website/versioned_docs/version-2.6.1/io-develop.md +++ b/site2/website/versioned_docs/version-2.6.1/io-develop.md @@ -8,15 +8,15 @@ original_id: io-develop This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -163,7 +163,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -192,7 +192,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). diff --git a/site2/website/versioned_docs/version-2.6.1/io-overview.md b/site2/website/versioned_docs/version-2.6.1/io-overview.md index 68df2472d6334..b538604edc46f 100644 --- a/site2/website/versioned_docs/version-2.6.1/io-overview.md +++ b/site2/website/versioned_docs/version-2.6.1/io-overview.md @@ -158,7 +158,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.6.1/io-quickstart.md b/site2/website/versioned_docs/version-2.6.1/io-quickstart.md index f32e2ec8e6d15..e08e598c308f2 100644 --- a/site2/website/versioned_docs/version-2.6.1/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.6.1/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you are able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you are able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However, all the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -109,7 +109,7 @@ This section demonstrates how to connect Pulsar to Cassandra. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink.md). ::: @@ -236,11 +236,11 @@ You can create a configuration file through one of the following methods. ``` -For more information, see [Cassandra sink connector](io-cassandra-sink). +For more information, see [Cassandra sink connector](io-cassandra-sink.md). ### Create a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on them. Run the following command to create a Cassandra sink connector with sink type _cassandra_ and the config file _examples/cassandra-sink.yml_ created previously. @@ -264,7 +264,7 @@ as a Pulsar Function and writes the messages produced in the topic _test_cassand ### Inspect a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * Get the information of a Cassandra sink. @@ -428,7 +428,7 @@ to monitor a connector and perform other operations on it. ### Delete a Cassandra Sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. ```bash @@ -451,7 +451,7 @@ This section demonstrates how to connect Pulsar to PostgreSQL. ::: ->For more information, see [JDBC sink connector](io-jdbc-sink). +>For more information, see [JDBC sink connector](io-jdbc-sink.md). ### Setup a PostgreSQL cluster @@ -611,7 +611,7 @@ In this section, you need to configure a JDBC sink connector. ### Create a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on it. This example creates a sink connector and specifies the desired information. @@ -657,7 +657,7 @@ The sink has been created successfully if the following message appears. ### Inspect a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * List all running JDBC sink(s). @@ -777,7 +777,7 @@ to monitor a connector and perform other operations on it. ### Stop a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to stop a connector and perform other operations on it. ```bash @@ -805,7 +805,7 @@ The sink instance has been stopped successfully if the following message disappe ### Restart a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to restart a connector and perform other operations on it. ```bash @@ -841,7 +841,7 @@ Note that `pulsar-admin sinks localrun options` **runs a sink connector locally* ### Update a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to update a connector and perform other operations on it. This example updates the parallelism of the _pulsar-postgres-jdbc-sink_ sink connector to 2. @@ -909,7 +909,7 @@ The result shows that the parallelism is 2. ### Delete a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. This example deletes the _pulsar-postgres-jdbc-sink_ sink connector. diff --git a/site2/website/versioned_docs/version-2.6.1/io-use.md b/site2/website/versioned_docs/version-2.6.1/io-use.md index 0992b7c918b99..058d7d366f484 100644 --- a/site2/website/versioned_docs/version-2.6.1/io-use.md +++ b/site2/website/versioned_docs/version-2.6.1/io-use.md @@ -15,7 +15,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.6.1/kubernetes-helm.md b/site2/website/versioned_docs/version-2.6.1/kubernetes-helm.md index a9f768e6a96b3..ea92a0968cd7d 100644 --- a/site2/website/versioned_docs/version-2.6.1/kubernetes-helm.md +++ b/site2/website/versioned_docs/version-2.6.1/kubernetes-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.6.1/pulsar-2.0.md b/site2/website/versioned_docs/version-2.6.1/pulsar-2.0.md index 11c5e66cb62c2..752590f57b558 100644 --- a/site2/website/versioned_docs/version-2.6.1/pulsar-2.0.md +++ b/site2/website/versioned_docs/version-2.6.1/pulsar-2.0.md @@ -5,13 +5,13 @@ sidebar_label: "Pulsar 2.0" original_id: pulsar-2.0 --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.6.1/reference-cli-tools.md b/site2/website/versioned_docs/version-2.6.1/reference-cli-tools.md index a46e5cc591c92..2f4229c0ba0e2 100644 --- a/site2/website/versioned_docs/version-2.6.1/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.6.1/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.6.1/schema-manage.md b/site2/website/versioned_docs/version-2.6.1/schema-manage.md index a4f0321bf86a9..cf706961b39ed 100644 --- a/site2/website/versioned_docs/version-2.6.1/schema-manage.md +++ b/site2/website/versioned_docs/version-2.6.1/schema-manage.md @@ -160,7 +160,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.6.1/schema-understand.md b/site2/website/versioned_docs/version-2.6.1/schema-understand.md index fda3d6dfecb4c..2b5fd1f44b360 100644 --- a/site2/website/versioned_docs/version-2.6.1/schema-understand.md +++ b/site2/website/versioned_docs/version-2.6.1/schema-understand.md @@ -372,7 +372,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```java diff --git a/site2/website/versioned_docs/version-2.6.1/security-athenz.md b/site2/website/versioned_docs/version-2.6.1/security-athenz.md index 947c3f470be46..8a39fe25316d0 100644 --- a/site2/website/versioned_docs/version-2.6.1/security-athenz.md +++ b/site2/website/versioned_docs/version-2.6.1/security-athenz.md @@ -76,7 +76,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.6.1/security-authorization.md b/site2/website/versioned_docs/version-2.6.1/security-authorization.md index 28200ad0744b9..ef78adc6cad0c 100644 --- a/site2/website/versioned_docs/version-2.6.1/security-authorization.md +++ b/site2/website/versioned_docs/version-2.6.1/security-authorization.md @@ -27,7 +27,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you can not only use superuser roles for administrators and clients but also for broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you can not only use superuser roles for administrators and clients but also for broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. The broker still checks the authorization of the request when the broker receives the forwarded request. @@ -58,7 +58,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -86,7 +86,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication diff --git a/site2/website/versioned_docs/version-2.6.1/security-bouncy-castle.md b/site2/website/versioned_docs/version-2.6.1/security-bouncy-castle.md index eab3ae64e08d5..534863744af9b 100644 --- a/site2/website/versioned_docs/version-2.6.1/security-bouncy-castle.md +++ b/site2/website/versioned_docs/version-2.6.1/security-bouncy-castle.md @@ -16,7 +16,7 @@ In Pulsar, security and crypto have dependencies on BouncyCastle Jars. For the d `Bouncy Castle` provides both [FIPS](https://www.bouncycastle.org/fips_faq.html) and non-FIPS version. But in a JVM, you can not include both of the 2 versions, and you need to exclude the current version before include the other. -In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. +In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption.md). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. ## Include dependencies of BC-non-FIPS diff --git a/site2/website/versioned_docs/version-2.6.1/security-extending.md b/site2/website/versioned_docs/version-2.6.1/security-extending.md index 57b4c6ad928ee..37930bb286576 100644 --- a/site2/website/versioned_docs/version-2.6.1/security-extending.md +++ b/site2/website/versioned_docs/version-2.6.1/security-extending.md @@ -9,7 +9,7 @@ Pulsar provides a way to use custom authentication and authorization mechanisms. ## Authentication -Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview). +Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview.md). You can choose to use a custom authentication mechanism by providing the implementation in the form of two plugins. One plugin is for the Client library and the other plugin is for the Pulsar Broker to validate the credentials. diff --git a/site2/website/versioned_docs/version-2.6.1/security-jwt.md b/site2/website/versioned_docs/version-2.6.1/security-jwt.md index 546e72449165a..17ea2cbf6228c 100644 --- a/site2/website/versioned_docs/version-2.6.1/security-jwt.md +++ b/site2/website/versioned_docs/version-2.6.1/security-jwt.md @@ -32,11 +32,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -303,7 +303,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.6.1/security-kerberos.md b/site2/website/versioned_docs/version-2.6.1/security-kerberos.md index 670586a9d1691..c49fa3bea1fce 100644 --- a/site2/website/versioned_docs/version-2.6.1/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.6.1/security-kerberos.md @@ -373,7 +373,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.6.1/security-overview.md b/site2/website/versioned_docs/version-2.6.1/security-overview.md index c7823104d145b..a1664fcb8b450 100644 --- a/site2/website/versioned_docs/version-2.6.1/security-overview.md +++ b/site2/website/versioned_docs/version-2.6.1/security-overview.md @@ -21,7 +21,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.6.1/security-tls-authentication.md b/site2/website/versioned_docs/version-2.6.1/security-tls-authentication.md index 20a6604a6c1fe..e1b67ffa05b78 100644 --- a/site2/website/versioned_docs/version-2.6.1/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.6.1/security-tls-authentication.md @@ -7,9 +7,9 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -101,7 +101,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -121,7 +121,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.6.1/security-tls-keystore.md b/site2/website/versioned_docs/version-2.6.1/security-tls-keystore.md index e6412ab6a86a7..180348ca708b8 100644 --- a/site2/website/versioned_docs/version-2.6.1/security-tls-keystore.md +++ b/site2/website/versioned_docs/version-2.6.1/security-tls-keystore.md @@ -7,7 +7,7 @@ original_id: security-tls-keystore ## Overview -Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication) between clients and Apache Pulsar service. +Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication.md) between clients and Apache Pulsar service. By default it uses PEM format file configuration. This page tries to describe use [KeyStore](https://en.wikipedia.org/wiki/Java_KeyStore) type configure for TLS. @@ -165,7 +165,7 @@ This is similar to [TLS encryption configuing for client with PEM type](security For a a minimal configuration, user need to provide the TrustStore information. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties @@ -258,7 +258,7 @@ webSocketServiceEnabled=false Besides the TLS encryption configuring. The main work is configuring the KeyStore, which contains a valid CN as client role, for client. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties diff --git a/site2/website/versioned_docs/version-2.6.1/security-tls-transport.md b/site2/website/versioned_docs/version-2.6.1/security-tls-transport.md index 9638b60eab1de..29fa49120c71e 100644 --- a/site2/website/versioned_docs/version-2.6.1/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.6.1/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,9 +27,9 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). -`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ## Create TLS certificates @@ -130,7 +130,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Broker Configuration -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -201,7 +201,7 @@ The examples below show hostname verification being disabled for the Java client ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.6.1/security-token-admin.md b/site2/website/versioned_docs/version-2.6.1/security-token-admin.md index 6bef23b106415..39bb6178e2bf3 100644 --- a/site2/website/versioned_docs/version-2.6.1/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.6.1/security-token-admin.md @@ -163,7 +163,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.6.1/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.6.1/sql-deployment-configurations.md index 28f2a3e7cb8a6..99215e81d021e 100644 --- a/site2/website/versioned_docs/version-2.6.1/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.6.1/sql-deployment-configurations.md @@ -54,7 +54,7 @@ Since Pulsar SQL is powered by [Presto](https://prestodb.io), the configuration :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.6.1/sql-overview.md b/site2/website/versioned_docs/version-2.6.1/sql-overview.md index 2bb78260265b2..8abb23ea2acbb 100644 --- a/site2/website/versioned_docs/version-2.6.1/sql-overview.md +++ b/site2/website/versioned_docs/version-2.6.1/sql-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Overview" original_id: sql-overview --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.6.1/standalone-docker.md b/site2/website/versioned_docs/version-2.6.1/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.6.1/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.6.1/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.6.1/standalone.md b/site2/website/versioned_docs/version-2.6.1/standalone.md index 220af52ce6861..e91f94140a4ac 100644 --- a/site2/website/versioned_docs/version-2.6.1/standalone.md +++ b/site2/website/versioned_docs/version-2.6.1/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -59,7 +59,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.6.2/admin-api-brokers.md b/site2/website/versioned_docs/version-2.6.2/admin-api-brokers.md index bdd6ff555ac74..9db8f153bf5da 100644 --- a/site2/website/versioned_docs/version-2.6.2/admin-api-brokers.md +++ b/site2/website/versioned_docs/version-2.6.2/admin-api-brokers.md @@ -12,7 +12,7 @@ Pulsar brokers consist of two components: [Brokers](reference-terminology.md#broker) can be managed via: -* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/brokers` endpoint of the admin {@inject: rest:REST:/} API * The `brokers` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin.html} object in the [Java API](client-libraries-java.md) @@ -95,7 +95,7 @@ One way to configure a Pulsar [broker](reference-terminology.md#broker) is to su But since all broker configuration in Pulsar is stored in ZooKeeper, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values. -* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. +* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. * In the Pulsar admin {@inject: rest:REST:/} API, dynamic configuration is managed through the `/admin/v2/brokers/configuration` endpoint. ### Update dynamic configuration diff --git a/site2/website/versioned_docs/version-2.6.2/admin-api-clusters.md b/site2/website/versioned_docs/version-2.6.2/admin-api-clusters.md index efd50a4c4b0c4..ac489a949d0f2 100644 --- a/site2/website/versioned_docs/version-2.6.2/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.6.2/admin-api-clusters.md @@ -10,7 +10,7 @@ servers (aka [bookies](reference-terminology.md#bookie)), and a [ZooKeeper](http Clusters can be managed via: -* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/clusters` endpoint of the admin {@inject: rest:REST:/} API * The `clusters` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md) @@ -62,7 +62,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > #### No cluster metadata initialization through the REST API or the Java admin API > @@ -86,11 +86,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. #### pulsar-admin diff --git a/site2/website/versioned_docs/version-2.6.2/admin-api-functions.md b/site2/website/versioned_docs/version-2.6.2/admin-api-functions.md index 96b7dc7ea52cd..315943492c214 100644 --- a/site2/website/versioned_docs/version-2.6.2/admin-api-functions.md +++ b/site2/website/versioned_docs/version-2.6.2/admin-api-functions.md @@ -15,9 +15,9 @@ Functions can be managed via the following methods. Method | Description ---|--- -**Admin CLI** | The [`functions`](reference-pulsar-admin.md#functions) command of the [`pulsar-admin`](reference-pulsar-admin) tool. +**Admin CLI** | The [`functions`](reference-pulsar-admin.md#functions) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool. **REST API** |The `/admin/v3/functions` endpoint of the admin {@inject: rest:REST:/} API. -**Java Admin API**| The `functions` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java). +**Java Admin API**| The `functions` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md). ## Function resources diff --git a/site2/website/versioned_docs/version-2.6.2/admin-api-namespaces.md b/site2/website/versioned_docs/version-2.6.2/admin-api-namespaces.md index f5ee8f44528ec..f48fe636b54a3 100644 --- a/site2/website/versioned_docs/version-2.6.2/admin-api-namespaces.md +++ b/site2/website/versioned_docs/version-2.6.2/admin-api-namespaces.md @@ -15,7 +15,7 @@ Pulsar [namespaces](reference-terminology.md#namespace) are logical groupings of Namespaces can be managed via: -* The [`namespaces`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`namespaces`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/namespaces` endpoint of the admin {@inject: rest:REST:/} API * The `namespaces` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md) diff --git a/site2/website/versioned_docs/version-2.6.2/admin-api-non-partitioned-topics.md b/site2/website/versioned_docs/version-2.6.2/admin-api-non-partitioned-topics.md index 9cd019d2abe8f..d34d8def6473f 100644 --- a/site2/website/versioned_docs/version-2.6.2/admin-api-non-partitioned-topics.md +++ b/site2/website/versioned_docs/version-2.6.2/admin-api-non-partitioned-topics.md @@ -6,7 +6,7 @@ original_id: admin-api-non-partitioned-topics --- -You can use Pulsar's [admin API](admin-api-overview) to create and manage non-partitioned topics. +You can use Pulsar's [admin API](admin-api-overview.md) to create and manage non-partitioned topics. In all of the instructions and commands below, the topic name structure is: diff --git a/site2/website/versioned_docs/version-2.6.2/admin-api-overview.md b/site2/website/versioned_docs/version-2.6.2/admin-api-overview.md index b6186cdd0ea9a..ec19fceda0db2 100644 --- a/site2/website/versioned_docs/version-2.6.2/admin-api-overview.md +++ b/site2/website/versioned_docs/version-2.6.2/admin-api-overview.md @@ -12,7 +12,7 @@ You can currently interact with the admin interface via: - Making HTTP calls against the admin {@inject: rest:REST:/} API provided by Pulsar [brokers](reference-terminology.md#broker). For some restful apis, they might be redirected to topic owner brokers for serving with [`307 Temporary Redirect`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/307), hence the HTTP callers should handle `307 Temporary Redirect`. If you are using `curl`, you should specify `-L` to handle redirections. -- The `pulsar-admin` CLI tool, which is available in the `bin` folder of your [Pulsar installation](getting-started-standalone): +- The `pulsar-admin` CLI tool, which is available in the `bin` folder of your [Pulsar installation](getting-started-standalone.md): ```shell @@ -20,7 +20,7 @@ $ bin/pulsar-admin ``` -Full documentation for this tool can be found in the [Pulsar command-line tools](reference-pulsar-admin) doc. +Full documentation for this tool can be found in the [Pulsar command-line tools](reference-pulsar-admin.md) doc. - A Java client interface. @@ -31,11 +31,11 @@ In this document, examples from each of the three available interfaces will be s ## Admin setup -Each of Pulsar's three admin interfaces---the [`pulsar-admin`](reference-pulsar-admin) CLI tool, the [Java admin API](/api/admin), and the {@inject: rest:REST:/} API ---requires some special setup if you have [authentication](security-overview.md#authentication-providers) enabled in your Pulsar [instance](reference-terminology.md#instance). +Each of Pulsar's three admin interfaces---the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool, the [Java admin API](/api/admin), and the {@inject: rest:REST:/} API ---requires some special setup if you have [authentication](security-overview.md#authentication-providers) enabled in your Pulsar [instance](reference-terminology.md#instance). ### pulsar-admin -If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: +If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin.md) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: |Name|Description|Default| |----|-----------|-------| diff --git a/site2/website/versioned_docs/version-2.6.2/admin-api-partitioned-topics.md b/site2/website/versioned_docs/version-2.6.2/admin-api-partitioned-topics.md index eef6efdf7407b..73edab60a4039 100644 --- a/site2/website/versioned_docs/version-2.6.2/admin-api-partitioned-topics.md +++ b/site2/website/versioned_docs/version-2.6.2/admin-api-partitioned-topics.md @@ -6,7 +6,7 @@ original_id: admin-api-partitioned-topics --- -You can use Pulsar's [admin API](admin-api-overview) to create and manage partitioned topics. +You can use Pulsar's [admin API](admin-api-overview.md) to create and manage partitioned topics. In all of the instructions and commands below, the topic name structure is: diff --git a/site2/website/versioned_docs/version-2.6.2/admin-api-tenants.md b/site2/website/versioned_docs/version-2.6.2/admin-api-tenants.md index 924cbce3d41ab..27764d6da3885 100644 --- a/site2/website/versioned_docs/version-2.6.2/admin-api-tenants.md +++ b/site2/website/versioned_docs/version-2.6.2/admin-api-tenants.md @@ -5,7 +5,7 @@ sidebar_label: "Tenants" original_id: admin-api-tenants --- -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters diff --git a/site2/website/versioned_docs/version-2.6.2/administration-dashboard.md b/site2/website/versioned_docs/version-2.6.2/administration-dashboard.md index 514b076239789..25f976609b40b 100644 --- a/site2/website/versioned_docs/version-2.6.2/administration-dashboard.md +++ b/site2/website/versioned_docs/version-2.6.2/administration-dashboard.md @@ -7,7 +7,7 @@ original_id: administration-dashboard :::note -Pulsar dashboard is deprecated. If you want to manage and monitor the stats of your topics, use [Pulsar Manager](administration-pulsar-manager). +Pulsar dashboard is deprecated. If you want to manage and monitor the stats of your topics, use [Pulsar Manager](administration-pulsar-manager.md). ::: diff --git a/site2/website/versioned_docs/version-2.6.2/administration-geo.md b/site2/website/versioned_docs/version-2.6.2/administration-geo.md index 42d32304a2fa4..4170d62f3ac9c 100644 --- a/site2/website/versioned_docs/version-2.6.2/administration-geo.md +++ b/site2/website/versioned_docs/version-2.6.2/administration-geo.md @@ -135,7 +135,7 @@ Once you create a geo-replication namespace, any topics that producers or consum By default, messages are replicated to all clusters configured for the namespace. You can restrict replication selectively by specifying a replication list for a message, and then that message is replicated only to the subset in the replication list. -The following is an example for the [Java API](client-libraries-java). Note the use of the `setReplicationClusters` method when you construct the {@inject: javadoc:Message:/client/org/apache/pulsar/client/api/Message} object: +The following is an example for the [Java API](client-libraries-java.md). Note the use of the `setReplicationClusters` method when you construct the {@inject: javadoc:Message:/client/org/apache/pulsar/client/api/Message} object: ```java @@ -157,7 +157,7 @@ producer.newMessage() #### Topic stats -Topic-specific statistics for geo-replication topics are available via the [`pulsar-admin`](reference-pulsar-admin) tool and {@inject: rest:REST:/} API: +Topic-specific statistics for geo-replication topics are available via the [`pulsar-admin`](reference-pulsar-admin.md) tool and {@inject: rest:REST:/} API: ```shell diff --git a/site2/website/versioned_docs/version-2.6.2/administration-zk-bk.md b/site2/website/versioned_docs/version-2.6.2/administration-zk-bk.md index 4d1038a416b9e..34cb025c7541c 100644 --- a/site2/website/versioned_docs/version-2.6.2/administration-zk-bk.md +++ b/site2/website/versioned_docs/version-2.6.2/administration-zk-bk.md @@ -174,7 +174,7 @@ The [`conf/global-zookeeper.conf`](reference-configuration.md#configuration-stor BookKeeper stores all durable messages in Pulsar. BookKeeper is a distributed [write-ahead log](https://en.wikipedia.org/wiki/Write-ahead_logging) WAL system that guarantees read consistency of independent message logs calls ledgers. Individual BookKeeper servers are also called *bookies*. -> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.6.2/client-libraries-cgo.md b/site2/website/versioned_docs/version-2.6.2/client-libraries-cgo.md index c79f7bb965473..f352f942b7714 100644 --- a/site2/website/versioned_docs/version-2.6.2/client-libraries-cgo.md +++ b/site2/website/versioned_docs/version-2.6.2/client-libraries-cgo.md @@ -24,7 +24,7 @@ Currently, the following Go clients are maintained in two repositories. ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Install go package @@ -57,7 +57,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -75,7 +75,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -236,14 +236,14 @@ Parameter | Description | Default `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | `Properties`| Attach a set of application defined properties to the producer. This properties will be visible in the topic stats | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | Set the number of max pending messages across all the partitions. This setting will be used to lower the max pending messages for each partition `MaxPendingMessages(int)`, if the total exceeds the configured value.| `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | `Batching` | Control whether automatic batching of messages is enabled for the producer. | false `BatchingMaxPublishDelay` | Set the time period within which the messages sent will be batched (default: 1ms) if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or until | 1ms `BatchingMaxMessages` | Set the maximum number of messages permitted in a batch. (default: 1000) If set to a value greater than 1, messages will be queued until this threshold is reached or batch interval has elapsed | 1000 @@ -512,7 +512,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.6.2/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.6.2/client-libraries-cpp.md index bb52d9effa404..684df5fd2d825 100644 --- a/site2/website/versioned_docs/version-2.6.2/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.6.2/client-libraries-cpp.md @@ -261,7 +261,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema -This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started). +This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started.md). ### Create producer with Avro schema diff --git a/site2/website/versioned_docs/version-2.6.2/client-libraries-dotnet.md b/site2/website/versioned_docs/version-2.6.2/client-libraries-dotnet.md index 4e0afe3d2ff6b..fbec5e473be69 100644 --- a/site2/website/versioned_docs/version-2.6.2/client-libraries-dotnet.md +++ b/site2/website/versioned_docs/version-2.6.2/client-libraries-dotnet.md @@ -170,7 +170,7 @@ var client = PulsarClient.Builder() Currently, the Pulsar C# client supports the TLS (Transport Layer Security) and JWT (JSON Web Token) authentication. -If you have followed [Authentication using TLS](security-tls-authentication), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: +If you have followed [Authentication using TLS](security-tls-authentication.md), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: 1. Create an unencrypted and password-less pfx file. diff --git a/site2/website/versioned_docs/version-2.6.2/client-libraries-go.md b/site2/website/versioned_docs/version-2.6.2/client-libraries-go.md index 97705a643c049..6c05e1d733422 100644 --- a/site2/website/versioned_docs/version-2.6.2/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.6.2/client-libraries-go.md @@ -35,7 +35,7 @@ import "github.com/apache/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -53,7 +53,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -680,7 +680,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker @@ -700,7 +700,7 @@ opts := pulsar.ClientOptions{ ## OAuth2 authentication -To use [OAuth2 authentication](security-oauth2), you'll need to configure your client to perform the following operations. +To use [OAuth2 authentication](security-oauth2.md), you'll need to configure your client to perform the following operations. This example shows how to configure OAuth2 authentication. ```go diff --git a/site2/website/versioned_docs/version-2.6.2/client-libraries-java.md b/site2/website/versioned_docs/version-2.6.2/client-libraries-java.md index 98572b1e35b60..c13fc69fff5f0 100644 --- a/site2/website/versioned_docs/version-2.6.2/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.6.2/client-libraries-java.md @@ -5,7 +5,7 @@ sidebar_label: "Java" original_id: client-libraries-java --- -You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), and [reader](#reader) of a Java client are thread-safe. @@ -14,10 +14,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Includes both `pulsar-client` and `pulsar-client-admin`

    Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.

    In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -57,7 +57,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -83,7 +83,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -114,7 +114,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -237,7 +237,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -782,7 +782,7 @@ Total hash range size is 65536, so the max end of the range should be less than ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -896,7 +896,7 @@ The following schema formats are currently available for Java: ## Authentication -Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2). You can use the Pulsar Java client with all of them. +Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2.md). You can use the Pulsar Java client with all of them. ### TLS Authentication @@ -923,7 +923,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` @@ -960,7 +960,7 @@ PulsarClient client = PulsarClient.builder() ### Oauth2 -The following example shows how to use [Oauth2](security-oauth2) as an authentication provider for the Pulsar Java client. +The following example shows how to use [Oauth2](security-oauth2.md) as an authentication provider for the Pulsar Java client. You can use the factory method to configure authentication for Pulsar Java client. diff --git a/site2/website/versioned_docs/version-2.6.2/client-libraries-node.md b/site2/website/versioned_docs/version-2.6.2/client-libraries-node.md index 8d7376d3793e5..b5a138cb86528 100644 --- a/site2/website/versioned_docs/version-2.6.2/client-libraries-node.md +++ b/site2/website/versioned_docs/version-2.6.2/client-libraries-node.md @@ -44,7 +44,7 @@ Also, this library works only in Node.js 10.x or later because it uses the [`nod ::: ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here is an example for `localhost`: @@ -62,7 +62,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication), the URL will look like something like this: +If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication.md), the URL will look like something like this: ```http @@ -97,7 +97,7 @@ The following configurable parameters are available for Pulsar clients: | Parameter | Description | Default | | :-------- | :---------- | :------ | | `serviceUrl` | The connection URL for the Pulsar cluster. See [above](#connection-urls) for more info. | | -| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication) for more info. | | +| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication.md) for more info. | | | `operationTimeoutSeconds` | The timeout for Node.js client operations (creating producers, subscribing to and unsubscribing from [topics](reference-terminology.md#topic)). Retries will occur until this threshold is reached, at which point the operation will fail. | 30 | | `ioThreads` | The number of threads to use for handling connections to Pulsar [brokers](reference-terminology.md#broker). | 1 | | `messageListenerThreads` | The number of threads used by message listeners ([consumers](#consumers) and [readers](#readers)). | 1 | @@ -147,7 +147,7 @@ Pulsar Node.js producers have the following methods available: | :-------- | :---------- | :------ | | `topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer publishes messages. The topic format is `` or `//`. For example, `sample/ns1/my-topic`. | | | `producerName` | A name for the producer. If you do not explicitly assign a name, Pulsar will automatically generate a globally unique name. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | | -| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30000 | +| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30000 | | `initialSequenceId` | The initial sequence ID of the message. When producer send message, add sequence ID to message. The ID is increased each time to send. | | | `maxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `send` method will fail *unless* `blockIfQueueFull` is set to `true`. | 1000 | | `maxPendingMessagesAcrossPartitions` | The maximum size of the sum of partition's pending queue. | 50000 | diff --git a/site2/website/versioned_docs/version-2.6.2/client-libraries-python.md b/site2/website/versioned_docs/version-2.6.2/client-libraries-python.md index 459345ddb7f5b..6d4c7e6a77888 100644 --- a/site2/website/versioned_docs/version-2.6.2/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.6.2/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. diff --git a/site2/website/versioned_docs/version-2.6.2/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.6.2/client-libraries-websocket.md index 9cc5d4acb8ded..f939a32d19b13 100644 --- a/site2/website/versioned_docs/version-2.6.2/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.6.2/client-libraries-websocket.md @@ -5,14 +5,14 @@ sidebar_label: "WebSocket" original_id: client-libraries-websocket --- -Pulsar's [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API is meant to provide a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSockets you can publish and consume messages and use all the features available in the [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp) client libraries. +Pulsar's [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API is meant to provide a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSockets you can publish and consume messages and use all the features available in the [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md) client libraries. > You can use Pulsar's WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.6.2/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.6.2/concepts-architecture-overview.md index 99fdc8de886d2..7dd93f7115eee 100644 --- a/site2/website/versioned_docs/version-2.6.2/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.6.2/concepts-architecture-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Architecture" original_id: concepts-architecture-overview --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -17,20 +17,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -40,9 +40,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -130,17 +130,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -148,7 +148,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.6.2/concepts-authentication.md b/site2/website/versioned_docs/version-2.6.2/concepts-authentication.md index 5b94e89e16934..d48e25a1dff9d 100644 --- a/site2/website/versioned_docs/version-2.6.2/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.6.2/concepts-authentication.md @@ -5,5 +5,5 @@ sidebar_label: "Authentication and Authorization" original_id: concepts-authentication --- -Pulsar supports a pluggable [authentication](security-overview) mechanism which can be configured at broker and it also supports authorization to identify client and its access rights on topics and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at broker and it also supports authorization to identify client and its access rights on topics and tenants. diff --git a/site2/website/versioned_docs/version-2.6.2/concepts-clients.md b/site2/website/versioned_docs/version-2.6.2/concepts-clients.md index 3492c8071ca68..6149b60bb4cb4 100644 --- a/site2/website/versioned_docs/version-2.6.2/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.6.2/concepts-clients.md @@ -5,7 +5,7 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. @@ -38,7 +38,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and will not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This will cause readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with have a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and will not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This will cause readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with have a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is just to allow users to know how behind the reader is and is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.6.2/concepts-messaging.md b/site2/website/versioned_docs/version-2.6.2/concepts-messaging.md index 29cebdfbaedf8..d8104060d987a 100644 --- a/site2/website/versioned_docs/version-2.6.2/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.6.2/concepts-messaging.md @@ -15,8 +15,8 @@ Messages are the basic "unit" of Pulsar. The following table lists the component Component | Description :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer who produces the message. If you do not specify a producer name, the default name is used. Sequence ID | Each Pulsar message belongs to an ordered sequence on its topic. The sequence ID of the message is its order in that sequence. @@ -24,7 +24,7 @@ Publish time | The timestamp of when the message is published. The timestamp is Event time | An optional timestamp attached to a message by applications. For example, applications attach a timestamp on when the message is processed. If nothing is set to event time, the value is `0`. TypedMessageBuilder | It is used to construct a message. You can set message properties such as the message key, message value with `TypedMessageBuilder`.
    When you set `TypedMessageBuilder`, set the key as a string. If you set the key as other types, for example, an AVRO object, the key is sent as bytes, and it is difficult to get the AVRO object back on the consumer. -> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol). +> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -101,7 +101,7 @@ Messages are received from [brokers](reference-terminology.md#broker) either syn ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -378,7 +378,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -390,7 +390,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -428,7 +428,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -483,7 +483,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -506,12 +506,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication.md). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.6.2/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.6.2/concepts-multi-tenancy.md index 685c3031d4aad..8cba09547d2d3 100644 --- a/site2/website/versioned_docs/version-2.6.2/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.6.2/concepts-multi-tenancy.md @@ -5,7 +5,7 @@ sidebar_label: "Multi Tenancy" original_id: concepts-multi-tenancy --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -21,7 +21,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -29,7 +29,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.6.2/concepts-overview.md b/site2/website/versioned_docs/version-2.6.2/concepts-overview.md index c1262b38daf16..e8a2f4b9d321a 100644 --- a/site2/website/versioned_docs/version-2.6.2/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.6.2/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.6.2/concepts-replication.md b/site2/website/versioned_docs/version-2.6.2/concepts-replication.md index 6e23962e99771..799f0eb4d92c6 100644 --- a/site2/website/versioned_docs/version-2.6.2/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.6.2/concepts-replication.md @@ -5,5 +5,5 @@ sidebar_label: "Geo Replication" original_id: concepts-replication --- -Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that. +Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo.md) in Pulsar enables you to do that. diff --git a/site2/website/versioned_docs/version-2.6.2/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.6.2/concepts-tiered-storage.md index 0b45b0a2ed501..f6988e53a8cd4 100644 --- a/site2/website/versioned_docs/version-2.6.2/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.6.2/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.6.2/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.6.2/concepts-topic-compaction.md index e402c965384fd..34b7ed7fbbd31 100644 --- a/site2/website/versioned_docs/version-2.6.2/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.6.2/concepts-topic-compaction.md @@ -7,7 +7,7 @@ original_id: concepts-topic-compaction Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -24,7 +24,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.6.2/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.6.2/cookbooks-compaction.md index f436568e393de..864fb75255bfa 100644 --- a/site2/website/versioned_docs/version-2.6.2/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.6.2/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.6.2/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.6.2/cookbooks-non-persistent.md index 8eb1307c3eae5..f831119681634 100644 --- a/site2/website/versioned_docs/version-2.6.2/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.6.2/cookbooks-non-persistent.md @@ -41,7 +41,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-non-persistent-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-non-persistent-topics.md) guide. ## Enabling @@ -55,7 +55,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.6.2/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.6.2/cookbooks-partitioned.md index 4649d84057c84..cdf3bf3d14005 100644 --- a/site2/website/versioned_docs/version-2.6.2/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.6.2/cookbooks-partitioned.md @@ -7,7 +7,7 @@ original_id: cookbooks-partitioned By default, Pulsar topics are served by a single broker. Using only a single broker limits a topic's maximum throughput. *Partitioned topics* are a special type of topic that can span multiple brokers and thus allow for much higher throughput. For an explanation of how partitioned topics work, see the [Partitioned Topics](concepts-messaging.md#partitioned-topics) concepts. -You can publish to partitioned topics using Pulsar client libraries and you can [create and manage](#managing-partitioned-topics) partitioned topics using Pulsar [admin API](admin-api-overview). +You can publish to partitioned topics using Pulsar client libraries and you can [create and manage](#managing-partitioned-topics) partitioned topics using Pulsar [admin API](admin-api-overview.md). ## Publish to partitioned topics @@ -100,4 +100,4 @@ If a message has a key, it supersedes the round robin routing policy. The follow ## Manage partitioned topics -You can use Pulsar [admin API](admin-api-overview.md) to create and manage [partitioned topics](admin-api-partitioned-topics). \ No newline at end of file +You can use Pulsar [admin API](admin-api-overview.md) to create and manage [partitioned topics](admin-api-partitioned-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.6.2/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.6.2/cookbooks-retention-expiry.md index 83f69f446ce05..45198cb6aab7c 100644 --- a/site2/website/versioned_docs/version-2.6.2/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.6.2/cookbooks-retention-expiry.md @@ -16,7 +16,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.6.2/deploy-aws.md b/site2/website/versioned_docs/version-2.6.2/deploy-aws.md index 542f40b1187ea..0ac377375f585 100644 --- a/site2/website/versioned_docs/version-2.6.2/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.6.2/deploy-aws.md @@ -5,7 +5,7 @@ sidebar_label: "Amazon Web Services" original_id: deploy-aws --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. diff --git a/site2/website/versioned_docs/version-2.6.2/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.6.2/deploy-bare-metal-multi-cluster.md index bdf8dfa251b3c..8d110c5d935bb 100644 --- a/site2/website/versioned_docs/version-2.6.2/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.6.2/deploy-bare-metal-multi-cluster.md @@ -9,17 +9,17 @@ original_id: deploy-bare-metal-multi-cluster 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using it in a startup or on a single team, you had better opt for a single cluster. For instructions on deploying a single cluster, -see the guide [here](deploy-bare-metal). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo). Deploying a multi-cluster Pulsar instance involves the following basic steps: +A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md). Deploying a multi-cluster Pulsar instance involves the following basic steps: * Deploying two separate [ZooKeeper](#deploy-zookeeper) quorums: a [local](#deploy-local-zookeeper) quorum for each cluster in the instance and a [configuration store](#configuration-store) quorum for instance-wide tasks * Initializing [cluster metadata](#cluster-metadata-initialization) for each cluster @@ -29,7 +29,7 @@ A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You If you want to deploy a single Pulsar cluster, see [Clusters and Brokers](getting-started-standalone.md#start-the-cluster). > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). ## System requirement Currently, Pulsar is available for 64-bit **macOS**, **Linux**, and **Windows**. To use Pulsar, you need to install 64-bit JRE/JDK 8 or later versions. @@ -67,7 +67,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -245,7 +245,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -364,17 +364,17 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). -You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > #### Service discovery already provided by many scheduling systems -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ### Service discovery setup -The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol) of Pulsar. +The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol.md) of Pulsar. To get started setting up the built-in service of discovery of Pulsar, you need to change a few parameters in the [`conf/discovery.conf`](reference-configuration.md#service-discovery) configuration file. Set the [`zookeeperServers`](reference-configuration.md#service-discovery-zookeeperServers) parameter to the ZooKeeper quorum connection string of the cluster and the [`configurationStoreServers`](reference-configuration.md#service-discovery-configurationStoreServers) setting to the [configuration store](reference-terminology.md#configuration-store) quorum connection string. @@ -399,7 +399,7 @@ $ bin/pulsar-daemon start discovery ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.6.2/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.6.2/deploy-bare-metal.md index 2c341748b5605..5895cfe566bad 100644 --- a/site2/website/versioned_docs/version-2.6.2/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.6.2/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using Pulsar in a startup or on a single team, it is simplest to opt for a single cluster. If you do need to run a multi-cluster Pulsar instance, -see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -101,7 +101,7 @@ The extracted directory contains the following subdirectories: Directory | Contains :---------|:-------- -`bin` |[command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` |[command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `data` | The data storage directory that ZooKeeper and BookKeeper use `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -263,9 +263,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (you had better not use a different port). -`--web-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). +`--web-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should not use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (you had better not use a different port). -`--broker-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). +`--broker-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). > If you do not have a DNS server, you can use multi-host format in the service URL with the following settings: @@ -389,7 +389,7 @@ webServicePortTls=8443 ### Enable Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -407,7 +407,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker). +If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker.md). ### Start Brokers diff --git a/site2/website/versioned_docs/version-2.6.2/deploy-dcos.md b/site2/website/versioned_docs/version-2.6.2/deploy-dcos.md index f5f8d1faa7dbe..952d5f47e30fa 100644 --- a/site2/website/versioned_docs/version-2.6.2/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.6.2/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -If you want to enable all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of +If you want to enable all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.6.2/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.6.2/deploy-kubernetes.md index dc7123d000467..1aefc6ad79f71 100644 --- a/site2/website/versioned_docs/version-2.6.2/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.6.2/deploy-kubernetes.md @@ -6,6 +6,6 @@ original_id: deploy-kubernetes --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.6.2/deploy-monitoring.md b/site2/website/versioned_docs/version-2.6.2/deploy-monitoring.md index 2efc328097c00..40e918d9434fd 100644 --- a/site2/website/versioned_docs/version-2.6.2/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.6.2/deploy-monitoring.md @@ -73,7 +73,7 @@ The default port for bookie is `8000` (instead of `8080`). You can change the po You can use Prometheus to collect and store the metrics data. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes that needs to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is automatically setup with the [provided](deploy-kubernetes) instructions. +When you run Pulsar on bare metal, you can provide the list of nodes that needs to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is automatically setup with the [provided](deploy-kubernetes.md) instructions. ## Dashboards @@ -83,7 +83,7 @@ For that reason you only need to collect time series of metrics aggregated at th ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Dashboard](administration-dashboard). +The per-topic dashboard instructions are available at [Dashboard](administration-dashboard.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.6.2/develop-schema.md b/site2/website/versioned_docs/version-2.6.2/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.6.2/develop-schema.md +++ b/site2/website/versioned_docs/version-2.6.2/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.6.2/functions-deploy.md b/site2/website/versioned_docs/version-2.6.2/functions-deploy.md index 51f1140693dbe..d9496b3ed5b48 100644 --- a/site2/website/versioned_docs/version-2.6.2/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.6.2/functions-deploy.md @@ -9,12 +9,12 @@ original_id: functions-deploy To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface diff --git a/site2/website/versioned_docs/version-2.6.2/functions-develop.md b/site2/website/versioned_docs/version-2.6.2/functions-develop.md index dce77b52bb914..ca885cfdc2b04 100644 --- a/site2/website/versioned_docs/version-2.6.2/functions-develop.md +++ b/site2/website/versioned_docs/version-2.6.2/functions-develop.md @@ -852,7 +852,7 @@ public class MetricRecorderFunction implements Function { ``` -> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring) guide. +> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring.md) guide.     @@ -883,11 +883,11 @@ Currently, the feature is not available in Go. ```` ### Access metrics -To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring) in Pulsar. +To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring.md) in Pulsar. ## Security -If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker). For more details, refer to [Security settings](functions-worker.md#security-settings). +If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings). Pulsar Functions can support the following providers: diff --git a/site2/website/versioned_docs/version-2.6.2/functions-overview.md b/site2/website/versioned_docs/version-2.6.2/functions-overview.md index 25dc6029acf70..816d301e0fd0e 100644 --- a/site2/website/versioned_docs/version-2.6.2/functions-overview.md +++ b/site2/website/versioned_docs/version-2.6.2/functions-overview.md @@ -164,7 +164,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.6.2/functions-worker.md b/site2/website/versioned_docs/version-2.6.2/functions-worker.md index e6a448955b4bf..2b8968a467648 100644 --- a/site2/website/versioned_docs/version-2.6.2/functions-worker.md +++ b/site2/website/versioned_docs/version-2.6.2/functions-worker.md @@ -4,7 +4,7 @@ title: Deploy and manage functions worker sidebar_label: "Setup: Pulsar Functions Worker" original_id: functions-worker --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -178,7 +178,7 @@ brokerClientTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -198,7 +198,7 @@ authenticationProviders: [ provider1, provider2 ] ``` For *TLS Authentication* provider, follow the example below to add the necessary settings. -See [TLS Authentication](security-tls-authentication) for more details. +See [TLS Authentication](security-tls-authentication.md) for more details. ``` @@ -222,7 +222,7 @@ properties: ``` For *Token Authentication* provider, add necessary settings for `properties` if needed. -See [Token Authentication](security-jwt) for more details. +See [Token Authentication](security-jwt.md) for more details. ``` diff --git a/site2/website/versioned_docs/version-2.6.2/getting-started-docker.md b/site2/website/versioned_docs/version-2.6.2/getting-started-docker.md index b9f12f0b52688..4f20971d75330 100644 --- a/site2/website/versioned_docs/version-2.6.2/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.6.2/getting-started-docker.md @@ -50,8 +50,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -110,7 +110,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.6.2/getting-started-helm.md b/site2/website/versioned_docs/version-2.6.2/getting-started-helm.md index cb4c6945f2022..d03266c1e814e 100644 --- a/site2/website/versioned_docs/version-2.6.2/getting-started-helm.md +++ b/site2/website/versioned_docs/version-2.6.2/getting-started-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -383,7 +383,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.6.2/getting-started-standalone.md b/site2/website/versioned_docs/version-2.6.2/getting-started-standalone.md index dad19d422f359..0921bb374b98b 100644 --- a/site2/website/versioned_docs/version-2.6.2/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.6.2/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -59,7 +59,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.6.2/helm-install.md b/site2/website/versioned_docs/version-2.6.2/helm-install.md index cb72f140967e9..4559b145d09a5 100644 --- a/site2/website/versioned_docs/version-2.6.2/helm-install.md +++ b/site2/website/versioned_docs/version-2.6.2/helm-install.md @@ -21,7 +21,7 @@ Before deploying Pulsar, you need to prepare your environment. ### Tools -Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools) on your computer. +Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools.md) on your computer. ## Cloud cluster preparation @@ -37,8 +37,8 @@ To create and connect to the Kubernetes cluster, follow the instructions: ## Pulsar deployment -Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy). +Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy.md). ## Pulsar upgrade -To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade). +To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade.md). diff --git a/site2/website/versioned_docs/version-2.6.2/helm-overview.md b/site2/website/versioned_docs/version-2.6.2/helm-overview.md index a77499e81ae7a..eb7b8d2ca91c7 100644 --- a/site2/website/versioned_docs/version-2.6.2/helm-overview.md +++ b/site2/website/versioned_docs/version-2.6.2/helm-overview.md @@ -53,9 +53,9 @@ It includes support for: ## Pulsar Helm chart quick start -To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install). +This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install.md). ## Troubleshooting @@ -65,7 +65,7 @@ We have done our best to make these charts as seamless as possible. Occasionally The Apache Pulsar Helm chart contains all required dependencies. -If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm) for your first iteration. +If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm.md) for your first iteration. 1. [Preparation](helm-prepare.md) 2. [Deployment](helm-deploy.md) @@ -83,7 +83,7 @@ helm upgrade pulsar -f pulsar.yaml ``` -For more detailed information, see [Upgrading](helm-upgrade). +For more detailed information, see [Upgrading](helm-upgrade.md). ## Uninstallation diff --git a/site2/website/versioned_docs/version-2.6.2/helm-prepare.md b/site2/website/versioned_docs/version-2.6.2/helm-prepare.md index 95d7e668f9eca..5e9f2f9ef4f68 100644 --- a/site2/website/versioned_docs/version-2.6.2/helm-prepare.md +++ b/site2/website/versioned_docs/version-2.6.2/helm-prepare.md @@ -89,4 +89,4 @@ PROJECT= USE_LOCAL_SSD=true LOCAL_SSD_COUNT= ## Next Steps -Continue with the [installation of the chart](helm-deploy) once you have the cluster up and running. +Continue with the [installation of the chart](helm-deploy.md) once you have the cluster up and running. diff --git a/site2/website/versioned_docs/version-2.6.2/helm-tools.md b/site2/website/versioned_docs/version-2.6.2/helm-tools.md index efd61eae32506..6ba89006913b6 100644 --- a/site2/website/versioned_docs/version-2.6.2/helm-tools.md +++ b/site2/website/versioned_docs/version-2.6.2/helm-tools.md @@ -25,7 +25,7 @@ You can get Helm from the project's [releases page](https://github.com/helm/helm ### Next steps -Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare). +Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare.md). ## Additional information diff --git a/site2/website/versioned_docs/version-2.6.2/helm-upgrade.md b/site2/website/versioned_docs/version-2.6.2/helm-upgrade.md index 3ae1cdf2ba2fd..18838f363fb5f 100644 --- a/site2/website/versioned_docs/version-2.6.2/helm-upgrade.md +++ b/site2/website/versioned_docs/version-2.6.2/helm-upgrade.md @@ -20,7 +20,7 @@ You can retrieve your previous `--set` arguments cleanly, with `helm get values To upgrade Apache Pulsar to a newer version, follow these steps: 1. Check the change log for the specific version you would like to upgrade to. -2. Go through [deployment documentation](helm-deploy) step by step. +2. Go through [deployment documentation](helm-deploy.md) step by step. 3. Extract your previous `--set` arguments with the following command. ```bash diff --git a/site2/website/versioned_docs/version-2.6.2/io-cdc.md b/site2/website/versioned_docs/version-2.6.2/io-cdc.md index 20f16ae0d6904..e6e662884826d 100644 --- a/site2/website/versioned_docs/version-2.6.2/io-cdc.md +++ b/site2/website/versioned_docs/version-2.6.2/io-cdc.md @@ -13,8 +13,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.6.2/io-develop.md b/site2/website/versioned_docs/version-2.6.2/io-develop.md index 4a815b3b9d6f6..7a356662560c9 100644 --- a/site2/website/versioned_docs/version-2.6.2/io-develop.md +++ b/site2/website/versioned_docs/version-2.6.2/io-develop.md @@ -8,15 +8,15 @@ original_id: io-develop This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -163,7 +163,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -192,7 +192,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). diff --git a/site2/website/versioned_docs/version-2.6.2/io-overview.md b/site2/website/versioned_docs/version-2.6.2/io-overview.md index 68df2472d6334..b538604edc46f 100644 --- a/site2/website/versioned_docs/version-2.6.2/io-overview.md +++ b/site2/website/versioned_docs/version-2.6.2/io-overview.md @@ -158,7 +158,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.6.2/io-quickstart.md b/site2/website/versioned_docs/version-2.6.2/io-quickstart.md index f32e2ec8e6d15..e08e598c308f2 100644 --- a/site2/website/versioned_docs/version-2.6.2/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.6.2/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you are able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you are able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However, all the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -109,7 +109,7 @@ This section demonstrates how to connect Pulsar to Cassandra. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink.md). ::: @@ -236,11 +236,11 @@ You can create a configuration file through one of the following methods. ``` -For more information, see [Cassandra sink connector](io-cassandra-sink). +For more information, see [Cassandra sink connector](io-cassandra-sink.md). ### Create a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on them. Run the following command to create a Cassandra sink connector with sink type _cassandra_ and the config file _examples/cassandra-sink.yml_ created previously. @@ -264,7 +264,7 @@ as a Pulsar Function and writes the messages produced in the topic _test_cassand ### Inspect a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * Get the information of a Cassandra sink. @@ -428,7 +428,7 @@ to monitor a connector and perform other operations on it. ### Delete a Cassandra Sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. ```bash @@ -451,7 +451,7 @@ This section demonstrates how to connect Pulsar to PostgreSQL. ::: ->For more information, see [JDBC sink connector](io-jdbc-sink). +>For more information, see [JDBC sink connector](io-jdbc-sink.md). ### Setup a PostgreSQL cluster @@ -611,7 +611,7 @@ In this section, you need to configure a JDBC sink connector. ### Create a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on it. This example creates a sink connector and specifies the desired information. @@ -657,7 +657,7 @@ The sink has been created successfully if the following message appears. ### Inspect a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * List all running JDBC sink(s). @@ -777,7 +777,7 @@ to monitor a connector and perform other operations on it. ### Stop a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to stop a connector and perform other operations on it. ```bash @@ -805,7 +805,7 @@ The sink instance has been stopped successfully if the following message disappe ### Restart a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to restart a connector and perform other operations on it. ```bash @@ -841,7 +841,7 @@ Note that `pulsar-admin sinks localrun options` **runs a sink connector locally* ### Update a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to update a connector and perform other operations on it. This example updates the parallelism of the _pulsar-postgres-jdbc-sink_ sink connector to 2. @@ -909,7 +909,7 @@ The result shows that the parallelism is 2. ### Delete a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. This example deletes the _pulsar-postgres-jdbc-sink_ sink connector. diff --git a/site2/website/versioned_docs/version-2.6.2/io-use.md b/site2/website/versioned_docs/version-2.6.2/io-use.md index 0992b7c918b99..058d7d366f484 100644 --- a/site2/website/versioned_docs/version-2.6.2/io-use.md +++ b/site2/website/versioned_docs/version-2.6.2/io-use.md @@ -15,7 +15,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.6.2/kubernetes-helm.md b/site2/website/versioned_docs/version-2.6.2/kubernetes-helm.md index a9f768e6a96b3..ea92a0968cd7d 100644 --- a/site2/website/versioned_docs/version-2.6.2/kubernetes-helm.md +++ b/site2/website/versioned_docs/version-2.6.2/kubernetes-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.6.2/pulsar-2.0.md b/site2/website/versioned_docs/version-2.6.2/pulsar-2.0.md index 11c5e66cb62c2..752590f57b558 100644 --- a/site2/website/versioned_docs/version-2.6.2/pulsar-2.0.md +++ b/site2/website/versioned_docs/version-2.6.2/pulsar-2.0.md @@ -5,13 +5,13 @@ sidebar_label: "Pulsar 2.0" original_id: pulsar-2.0 --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.6.2/reference-cli-tools.md b/site2/website/versioned_docs/version-2.6.2/reference-cli-tools.md index 9c6ff2031b8a8..cddcf9f013b4a 100644 --- a/site2/website/versioned_docs/version-2.6.2/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.6.2/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.6.2/schema-manage.md b/site2/website/versioned_docs/version-2.6.2/schema-manage.md index 662b8b523726d..84d0ccdc23164 100644 --- a/site2/website/versioned_docs/version-2.6.2/schema-manage.md +++ b/site2/website/versioned_docs/version-2.6.2/schema-manage.md @@ -160,7 +160,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.6.2/schema-understand.md b/site2/website/versioned_docs/version-2.6.2/schema-understand.md index fda3d6dfecb4c..2b5fd1f44b360 100644 --- a/site2/website/versioned_docs/version-2.6.2/schema-understand.md +++ b/site2/website/versioned_docs/version-2.6.2/schema-understand.md @@ -372,7 +372,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```java diff --git a/site2/website/versioned_docs/version-2.6.2/security-athenz.md b/site2/website/versioned_docs/version-2.6.2/security-athenz.md index 947c3f470be46..8a39fe25316d0 100644 --- a/site2/website/versioned_docs/version-2.6.2/security-athenz.md +++ b/site2/website/versioned_docs/version-2.6.2/security-athenz.md @@ -76,7 +76,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.6.2/security-authorization.md b/site2/website/versioned_docs/version-2.6.2/security-authorization.md index 28200ad0744b9..ef78adc6cad0c 100644 --- a/site2/website/versioned_docs/version-2.6.2/security-authorization.md +++ b/site2/website/versioned_docs/version-2.6.2/security-authorization.md @@ -27,7 +27,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you can not only use superuser roles for administrators and clients but also for broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you can not only use superuser roles for administrators and clients but also for broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. The broker still checks the authorization of the request when the broker receives the forwarded request. @@ -58,7 +58,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -86,7 +86,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication diff --git a/site2/website/versioned_docs/version-2.6.2/security-bouncy-castle.md b/site2/website/versioned_docs/version-2.6.2/security-bouncy-castle.md index d012a7790a8ab..e8729a1188952 100644 --- a/site2/website/versioned_docs/version-2.6.2/security-bouncy-castle.md +++ b/site2/website/versioned_docs/version-2.6.2/security-bouncy-castle.md @@ -16,7 +16,7 @@ In Pulsar, security and crypto have dependencies on BouncyCastle Jars. For the d `Bouncy Castle` provides both [FIPS](https://www.bouncycastle.org/fips_faq.html) and non-FIPS version. But in a JVM, you can not include both of the 2 versions, and you need to exclude the current version before include the other. -In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. +In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption.md). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. ## Include dependencies of BC-non-FIPS diff --git a/site2/website/versioned_docs/version-2.6.2/security-extending.md b/site2/website/versioned_docs/version-2.6.2/security-extending.md index 57b4c6ad928ee..37930bb286576 100644 --- a/site2/website/versioned_docs/version-2.6.2/security-extending.md +++ b/site2/website/versioned_docs/version-2.6.2/security-extending.md @@ -9,7 +9,7 @@ Pulsar provides a way to use custom authentication and authorization mechanisms. ## Authentication -Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview). +Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview.md). You can choose to use a custom authentication mechanism by providing the implementation in the form of two plugins. One plugin is for the Client library and the other plugin is for the Pulsar Broker to validate the credentials. diff --git a/site2/website/versioned_docs/version-2.6.2/security-jwt.md b/site2/website/versioned_docs/version-2.6.2/security-jwt.md index 546e72449165a..17ea2cbf6228c 100644 --- a/site2/website/versioned_docs/version-2.6.2/security-jwt.md +++ b/site2/website/versioned_docs/version-2.6.2/security-jwt.md @@ -32,11 +32,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -303,7 +303,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.6.2/security-kerberos.md b/site2/website/versioned_docs/version-2.6.2/security-kerberos.md index 670586a9d1691..c49fa3bea1fce 100644 --- a/site2/website/versioned_docs/version-2.6.2/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.6.2/security-kerberos.md @@ -373,7 +373,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.6.2/security-overview.md b/site2/website/versioned_docs/version-2.6.2/security-overview.md index c7823104d145b..a1664fcb8b450 100644 --- a/site2/website/versioned_docs/version-2.6.2/security-overview.md +++ b/site2/website/versioned_docs/version-2.6.2/security-overview.md @@ -21,7 +21,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.6.2/security-tls-authentication.md b/site2/website/versioned_docs/version-2.6.2/security-tls-authentication.md index 20a6604a6c1fe..e1b67ffa05b78 100644 --- a/site2/website/versioned_docs/version-2.6.2/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.6.2/security-tls-authentication.md @@ -7,9 +7,9 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -101,7 +101,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -121,7 +121,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.6.2/security-tls-keystore.md b/site2/website/versioned_docs/version-2.6.2/security-tls-keystore.md index e6412ab6a86a7..180348ca708b8 100644 --- a/site2/website/versioned_docs/version-2.6.2/security-tls-keystore.md +++ b/site2/website/versioned_docs/version-2.6.2/security-tls-keystore.md @@ -7,7 +7,7 @@ original_id: security-tls-keystore ## Overview -Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication) between clients and Apache Pulsar service. +Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication.md) between clients and Apache Pulsar service. By default it uses PEM format file configuration. This page tries to describe use [KeyStore](https://en.wikipedia.org/wiki/Java_KeyStore) type configure for TLS. @@ -165,7 +165,7 @@ This is similar to [TLS encryption configuing for client with PEM type](security For a a minimal configuration, user need to provide the TrustStore information. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties @@ -258,7 +258,7 @@ webSocketServiceEnabled=false Besides the TLS encryption configuring. The main work is configuring the KeyStore, which contains a valid CN as client role, for client. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties diff --git a/site2/website/versioned_docs/version-2.6.2/security-tls-transport.md b/site2/website/versioned_docs/version-2.6.2/security-tls-transport.md index 9638b60eab1de..29fa49120c71e 100644 --- a/site2/website/versioned_docs/version-2.6.2/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.6.2/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,9 +27,9 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). -`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ## Create TLS certificates @@ -130,7 +130,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Broker Configuration -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -201,7 +201,7 @@ The examples below show hostname verification being disabled for the Java client ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.6.2/security-token-admin.md b/site2/website/versioned_docs/version-2.6.2/security-token-admin.md index 6bef23b106415..39bb6178e2bf3 100644 --- a/site2/website/versioned_docs/version-2.6.2/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.6.2/security-token-admin.md @@ -163,7 +163,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.6.2/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.6.2/sql-deployment-configurations.md index 28f2a3e7cb8a6..99215e81d021e 100644 --- a/site2/website/versioned_docs/version-2.6.2/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.6.2/sql-deployment-configurations.md @@ -54,7 +54,7 @@ Since Pulsar SQL is powered by [Presto](https://prestodb.io), the configuration :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.6.2/sql-overview.md b/site2/website/versioned_docs/version-2.6.2/sql-overview.md index 2bb78260265b2..8abb23ea2acbb 100644 --- a/site2/website/versioned_docs/version-2.6.2/sql-overview.md +++ b/site2/website/versioned_docs/version-2.6.2/sql-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Overview" original_id: sql-overview --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.6.2/standalone-docker.md b/site2/website/versioned_docs/version-2.6.2/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.6.2/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.6.2/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.6.2/standalone.md b/site2/website/versioned_docs/version-2.6.2/standalone.md index 220af52ce6861..e91f94140a4ac 100644 --- a/site2/website/versioned_docs/version-2.6.2/standalone.md +++ b/site2/website/versioned_docs/version-2.6.2/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -59,7 +59,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.6.3/admin-api-brokers.md b/site2/website/versioned_docs/version-2.6.3/admin-api-brokers.md index bdd6ff555ac74..9db8f153bf5da 100644 --- a/site2/website/versioned_docs/version-2.6.3/admin-api-brokers.md +++ b/site2/website/versioned_docs/version-2.6.3/admin-api-brokers.md @@ -12,7 +12,7 @@ Pulsar brokers consist of two components: [Brokers](reference-terminology.md#broker) can be managed via: -* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/brokers` endpoint of the admin {@inject: rest:REST:/} API * The `brokers` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin.html} object in the [Java API](client-libraries-java.md) @@ -95,7 +95,7 @@ One way to configure a Pulsar [broker](reference-terminology.md#broker) is to su But since all broker configuration in Pulsar is stored in ZooKeeper, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values. -* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. +* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. * In the Pulsar admin {@inject: rest:REST:/} API, dynamic configuration is managed through the `/admin/v2/brokers/configuration` endpoint. ### Update dynamic configuration diff --git a/site2/website/versioned_docs/version-2.6.3/admin-api-clusters.md b/site2/website/versioned_docs/version-2.6.3/admin-api-clusters.md index efd50a4c4b0c4..ac489a949d0f2 100644 --- a/site2/website/versioned_docs/version-2.6.3/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.6.3/admin-api-clusters.md @@ -10,7 +10,7 @@ servers (aka [bookies](reference-terminology.md#bookie)), and a [ZooKeeper](http Clusters can be managed via: -* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/clusters` endpoint of the admin {@inject: rest:REST:/} API * The `clusters` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md) @@ -62,7 +62,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > #### No cluster metadata initialization through the REST API or the Java admin API > @@ -86,11 +86,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. #### pulsar-admin diff --git a/site2/website/versioned_docs/version-2.6.3/admin-api-functions.md b/site2/website/versioned_docs/version-2.6.3/admin-api-functions.md index 96b7dc7ea52cd..315943492c214 100644 --- a/site2/website/versioned_docs/version-2.6.3/admin-api-functions.md +++ b/site2/website/versioned_docs/version-2.6.3/admin-api-functions.md @@ -15,9 +15,9 @@ Functions can be managed via the following methods. Method | Description ---|--- -**Admin CLI** | The [`functions`](reference-pulsar-admin.md#functions) command of the [`pulsar-admin`](reference-pulsar-admin) tool. +**Admin CLI** | The [`functions`](reference-pulsar-admin.md#functions) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool. **REST API** |The `/admin/v3/functions` endpoint of the admin {@inject: rest:REST:/} API. -**Java Admin API**| The `functions` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java). +**Java Admin API**| The `functions` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md). ## Function resources diff --git a/site2/website/versioned_docs/version-2.6.3/admin-api-namespaces.md b/site2/website/versioned_docs/version-2.6.3/admin-api-namespaces.md index f5ee8f44528ec..f48fe636b54a3 100644 --- a/site2/website/versioned_docs/version-2.6.3/admin-api-namespaces.md +++ b/site2/website/versioned_docs/version-2.6.3/admin-api-namespaces.md @@ -15,7 +15,7 @@ Pulsar [namespaces](reference-terminology.md#namespace) are logical groupings of Namespaces can be managed via: -* The [`namespaces`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`namespaces`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/namespaces` endpoint of the admin {@inject: rest:REST:/} API * The `namespaces` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md) diff --git a/site2/website/versioned_docs/version-2.6.3/admin-api-non-partitioned-topics.md b/site2/website/versioned_docs/version-2.6.3/admin-api-non-partitioned-topics.md index 9cd019d2abe8f..d34d8def6473f 100644 --- a/site2/website/versioned_docs/version-2.6.3/admin-api-non-partitioned-topics.md +++ b/site2/website/versioned_docs/version-2.6.3/admin-api-non-partitioned-topics.md @@ -6,7 +6,7 @@ original_id: admin-api-non-partitioned-topics --- -You can use Pulsar's [admin API](admin-api-overview) to create and manage non-partitioned topics. +You can use Pulsar's [admin API](admin-api-overview.md) to create and manage non-partitioned topics. In all of the instructions and commands below, the topic name structure is: diff --git a/site2/website/versioned_docs/version-2.6.3/admin-api-overview.md b/site2/website/versioned_docs/version-2.6.3/admin-api-overview.md index b6186cdd0ea9a..ec19fceda0db2 100644 --- a/site2/website/versioned_docs/version-2.6.3/admin-api-overview.md +++ b/site2/website/versioned_docs/version-2.6.3/admin-api-overview.md @@ -12,7 +12,7 @@ You can currently interact with the admin interface via: - Making HTTP calls against the admin {@inject: rest:REST:/} API provided by Pulsar [brokers](reference-terminology.md#broker). For some restful apis, they might be redirected to topic owner brokers for serving with [`307 Temporary Redirect`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/307), hence the HTTP callers should handle `307 Temporary Redirect`. If you are using `curl`, you should specify `-L` to handle redirections. -- The `pulsar-admin` CLI tool, which is available in the `bin` folder of your [Pulsar installation](getting-started-standalone): +- The `pulsar-admin` CLI tool, which is available in the `bin` folder of your [Pulsar installation](getting-started-standalone.md): ```shell @@ -20,7 +20,7 @@ $ bin/pulsar-admin ``` -Full documentation for this tool can be found in the [Pulsar command-line tools](reference-pulsar-admin) doc. +Full documentation for this tool can be found in the [Pulsar command-line tools](reference-pulsar-admin.md) doc. - A Java client interface. @@ -31,11 +31,11 @@ In this document, examples from each of the three available interfaces will be s ## Admin setup -Each of Pulsar's three admin interfaces---the [`pulsar-admin`](reference-pulsar-admin) CLI tool, the [Java admin API](/api/admin), and the {@inject: rest:REST:/} API ---requires some special setup if you have [authentication](security-overview.md#authentication-providers) enabled in your Pulsar [instance](reference-terminology.md#instance). +Each of Pulsar's three admin interfaces---the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool, the [Java admin API](/api/admin), and the {@inject: rest:REST:/} API ---requires some special setup if you have [authentication](security-overview.md#authentication-providers) enabled in your Pulsar [instance](reference-terminology.md#instance). ### pulsar-admin -If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: +If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin.md) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: |Name|Description|Default| |----|-----------|-------| diff --git a/site2/website/versioned_docs/version-2.6.3/admin-api-partitioned-topics.md b/site2/website/versioned_docs/version-2.6.3/admin-api-partitioned-topics.md index eef6efdf7407b..73edab60a4039 100644 --- a/site2/website/versioned_docs/version-2.6.3/admin-api-partitioned-topics.md +++ b/site2/website/versioned_docs/version-2.6.3/admin-api-partitioned-topics.md @@ -6,7 +6,7 @@ original_id: admin-api-partitioned-topics --- -You can use Pulsar's [admin API](admin-api-overview) to create and manage partitioned topics. +You can use Pulsar's [admin API](admin-api-overview.md) to create and manage partitioned topics. In all of the instructions and commands below, the topic name structure is: diff --git a/site2/website/versioned_docs/version-2.6.3/admin-api-tenants.md b/site2/website/versioned_docs/version-2.6.3/admin-api-tenants.md index 924cbce3d41ab..27764d6da3885 100644 --- a/site2/website/versioned_docs/version-2.6.3/admin-api-tenants.md +++ b/site2/website/versioned_docs/version-2.6.3/admin-api-tenants.md @@ -5,7 +5,7 @@ sidebar_label: "Tenants" original_id: admin-api-tenants --- -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters diff --git a/site2/website/versioned_docs/version-2.6.3/administration-dashboard.md b/site2/website/versioned_docs/version-2.6.3/administration-dashboard.md index 514b076239789..25f976609b40b 100644 --- a/site2/website/versioned_docs/version-2.6.3/administration-dashboard.md +++ b/site2/website/versioned_docs/version-2.6.3/administration-dashboard.md @@ -7,7 +7,7 @@ original_id: administration-dashboard :::note -Pulsar dashboard is deprecated. If you want to manage and monitor the stats of your topics, use [Pulsar Manager](administration-pulsar-manager). +Pulsar dashboard is deprecated. If you want to manage and monitor the stats of your topics, use [Pulsar Manager](administration-pulsar-manager.md). ::: diff --git a/site2/website/versioned_docs/version-2.6.3/administration-geo.md b/site2/website/versioned_docs/version-2.6.3/administration-geo.md index 42d32304a2fa4..4170d62f3ac9c 100644 --- a/site2/website/versioned_docs/version-2.6.3/administration-geo.md +++ b/site2/website/versioned_docs/version-2.6.3/administration-geo.md @@ -135,7 +135,7 @@ Once you create a geo-replication namespace, any topics that producers or consum By default, messages are replicated to all clusters configured for the namespace. You can restrict replication selectively by specifying a replication list for a message, and then that message is replicated only to the subset in the replication list. -The following is an example for the [Java API](client-libraries-java). Note the use of the `setReplicationClusters` method when you construct the {@inject: javadoc:Message:/client/org/apache/pulsar/client/api/Message} object: +The following is an example for the [Java API](client-libraries-java.md). Note the use of the `setReplicationClusters` method when you construct the {@inject: javadoc:Message:/client/org/apache/pulsar/client/api/Message} object: ```java @@ -157,7 +157,7 @@ producer.newMessage() #### Topic stats -Topic-specific statistics for geo-replication topics are available via the [`pulsar-admin`](reference-pulsar-admin) tool and {@inject: rest:REST:/} API: +Topic-specific statistics for geo-replication topics are available via the [`pulsar-admin`](reference-pulsar-admin.md) tool and {@inject: rest:REST:/} API: ```shell diff --git a/site2/website/versioned_docs/version-2.6.3/administration-zk-bk.md b/site2/website/versioned_docs/version-2.6.3/administration-zk-bk.md index 5daa66f0d0127..2c080123ca81d 100644 --- a/site2/website/versioned_docs/version-2.6.3/administration-zk-bk.md +++ b/site2/website/versioned_docs/version-2.6.3/administration-zk-bk.md @@ -175,7 +175,7 @@ The [`conf/global-zookeeper.conf`](reference-configuration.md#configuration-stor BookKeeper stores all durable messages in Pulsar. BookKeeper is a distributed [write-ahead log](https://en.wikipedia.org/wiki/Write-ahead_logging) WAL system that guarantees read consistency of independent message logs calls ledgers. Individual BookKeeper servers are also called *bookies*. -> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.6.3/client-libraries-cgo.md b/site2/website/versioned_docs/version-2.6.3/client-libraries-cgo.md index c79f7bb965473..f352f942b7714 100644 --- a/site2/website/versioned_docs/version-2.6.3/client-libraries-cgo.md +++ b/site2/website/versioned_docs/version-2.6.3/client-libraries-cgo.md @@ -24,7 +24,7 @@ Currently, the following Go clients are maintained in two repositories. ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Install go package @@ -57,7 +57,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -75,7 +75,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -236,14 +236,14 @@ Parameter | Description | Default `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | `Properties`| Attach a set of application defined properties to the producer. This properties will be visible in the topic stats | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | Set the number of max pending messages across all the partitions. This setting will be used to lower the max pending messages for each partition `MaxPendingMessages(int)`, if the total exceeds the configured value.| `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | `Batching` | Control whether automatic batching of messages is enabled for the producer. | false `BatchingMaxPublishDelay` | Set the time period within which the messages sent will be batched (default: 1ms) if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or until | 1ms `BatchingMaxMessages` | Set the maximum number of messages permitted in a batch. (default: 1000) If set to a value greater than 1, messages will be queued until this threshold is reached or batch interval has elapsed | 1000 @@ -512,7 +512,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.6.3/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.6.3/client-libraries-cpp.md index bb52d9effa404..684df5fd2d825 100644 --- a/site2/website/versioned_docs/version-2.6.3/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.6.3/client-libraries-cpp.md @@ -261,7 +261,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema -This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started). +This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started.md). ### Create producer with Avro schema diff --git a/site2/website/versioned_docs/version-2.6.3/client-libraries-dotnet.md b/site2/website/versioned_docs/version-2.6.3/client-libraries-dotnet.md index ade664cd95277..b574fa0b2e5ed 100644 --- a/site2/website/versioned_docs/version-2.6.3/client-libraries-dotnet.md +++ b/site2/website/versioned_docs/version-2.6.3/client-libraries-dotnet.md @@ -170,7 +170,7 @@ var client = PulsarClient.Builder() Currently, the Pulsar C# client supports the TLS (Transport Layer Security) and JWT (JSON Web Token) authentication. -If you have followed [Authentication using TLS](security-tls-authentication), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: +If you have followed [Authentication using TLS](security-tls-authentication.md), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: 1. Create an unencrypted and password-less pfx file. diff --git a/site2/website/versioned_docs/version-2.6.3/client-libraries-go.md b/site2/website/versioned_docs/version-2.6.3/client-libraries-go.md index 97705a643c049..6c05e1d733422 100644 --- a/site2/website/versioned_docs/version-2.6.3/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.6.3/client-libraries-go.md @@ -35,7 +35,7 @@ import "github.com/apache/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -53,7 +53,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -680,7 +680,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker @@ -700,7 +700,7 @@ opts := pulsar.ClientOptions{ ## OAuth2 authentication -To use [OAuth2 authentication](security-oauth2), you'll need to configure your client to perform the following operations. +To use [OAuth2 authentication](security-oauth2.md), you'll need to configure your client to perform the following operations. This example shows how to configure OAuth2 authentication. ```go diff --git a/site2/website/versioned_docs/version-2.6.3/client-libraries-java.md b/site2/website/versioned_docs/version-2.6.3/client-libraries-java.md index 98572b1e35b60..c13fc69fff5f0 100644 --- a/site2/website/versioned_docs/version-2.6.3/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.6.3/client-libraries-java.md @@ -5,7 +5,7 @@ sidebar_label: "Java" original_id: client-libraries-java --- -You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), and [reader](#reader) of a Java client are thread-safe. @@ -14,10 +14,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Includes both `pulsar-client` and `pulsar-client-admin`

    Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.

    In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -57,7 +57,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -83,7 +83,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -114,7 +114,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -237,7 +237,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -782,7 +782,7 @@ Total hash range size is 65536, so the max end of the range should be less than ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -896,7 +896,7 @@ The following schema formats are currently available for Java: ## Authentication -Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2). You can use the Pulsar Java client with all of them. +Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2.md). You can use the Pulsar Java client with all of them. ### TLS Authentication @@ -923,7 +923,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` @@ -960,7 +960,7 @@ PulsarClient client = PulsarClient.builder() ### Oauth2 -The following example shows how to use [Oauth2](security-oauth2) as an authentication provider for the Pulsar Java client. +The following example shows how to use [Oauth2](security-oauth2.md) as an authentication provider for the Pulsar Java client. You can use the factory method to configure authentication for Pulsar Java client. diff --git a/site2/website/versioned_docs/version-2.6.3/client-libraries-node.md b/site2/website/versioned_docs/version-2.6.3/client-libraries-node.md index 8d7376d3793e5..b5a138cb86528 100644 --- a/site2/website/versioned_docs/version-2.6.3/client-libraries-node.md +++ b/site2/website/versioned_docs/version-2.6.3/client-libraries-node.md @@ -44,7 +44,7 @@ Also, this library works only in Node.js 10.x or later because it uses the [`nod ::: ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here is an example for `localhost`: @@ -62,7 +62,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication), the URL will look like something like this: +If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication.md), the URL will look like something like this: ```http @@ -97,7 +97,7 @@ The following configurable parameters are available for Pulsar clients: | Parameter | Description | Default | | :-------- | :---------- | :------ | | `serviceUrl` | The connection URL for the Pulsar cluster. See [above](#connection-urls) for more info. | | -| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication) for more info. | | +| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication.md) for more info. | | | `operationTimeoutSeconds` | The timeout for Node.js client operations (creating producers, subscribing to and unsubscribing from [topics](reference-terminology.md#topic)). Retries will occur until this threshold is reached, at which point the operation will fail. | 30 | | `ioThreads` | The number of threads to use for handling connections to Pulsar [brokers](reference-terminology.md#broker). | 1 | | `messageListenerThreads` | The number of threads used by message listeners ([consumers](#consumers) and [readers](#readers)). | 1 | @@ -147,7 +147,7 @@ Pulsar Node.js producers have the following methods available: | :-------- | :---------- | :------ | | `topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer publishes messages. The topic format is `` or `//`. For example, `sample/ns1/my-topic`. | | | `producerName` | A name for the producer. If you do not explicitly assign a name, Pulsar will automatically generate a globally unique name. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | | -| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30000 | +| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30000 | | `initialSequenceId` | The initial sequence ID of the message. When producer send message, add sequence ID to message. The ID is increased each time to send. | | | `maxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `send` method will fail *unless* `blockIfQueueFull` is set to `true`. | 1000 | | `maxPendingMessagesAcrossPartitions` | The maximum size of the sum of partition's pending queue. | 50000 | diff --git a/site2/website/versioned_docs/version-2.6.3/client-libraries-python.md b/site2/website/versioned_docs/version-2.6.3/client-libraries-python.md index 459345ddb7f5b..6d4c7e6a77888 100644 --- a/site2/website/versioned_docs/version-2.6.3/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.6.3/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. diff --git a/site2/website/versioned_docs/version-2.6.3/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.6.3/client-libraries-websocket.md index 9cc5d4acb8ded..f939a32d19b13 100644 --- a/site2/website/versioned_docs/version-2.6.3/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.6.3/client-libraries-websocket.md @@ -5,14 +5,14 @@ sidebar_label: "WebSocket" original_id: client-libraries-websocket --- -Pulsar's [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API is meant to provide a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSockets you can publish and consume messages and use all the features available in the [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp) client libraries. +Pulsar's [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API is meant to provide a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSockets you can publish and consume messages and use all the features available in the [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md) client libraries. > You can use Pulsar's WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.6.3/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.6.3/concepts-architecture-overview.md index 99fdc8de886d2..7dd93f7115eee 100644 --- a/site2/website/versioned_docs/version-2.6.3/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.6.3/concepts-architecture-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Architecture" original_id: concepts-architecture-overview --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -17,20 +17,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -40,9 +40,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -130,17 +130,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -148,7 +148,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.6.3/concepts-authentication.md b/site2/website/versioned_docs/version-2.6.3/concepts-authentication.md index 5b94e89e16934..d48e25a1dff9d 100644 --- a/site2/website/versioned_docs/version-2.6.3/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.6.3/concepts-authentication.md @@ -5,5 +5,5 @@ sidebar_label: "Authentication and Authorization" original_id: concepts-authentication --- -Pulsar supports a pluggable [authentication](security-overview) mechanism which can be configured at broker and it also supports authorization to identify client and its access rights on topics and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at broker and it also supports authorization to identify client and its access rights on topics and tenants. diff --git a/site2/website/versioned_docs/version-2.6.3/concepts-clients.md b/site2/website/versioned_docs/version-2.6.3/concepts-clients.md index 3492c8071ca68..6149b60bb4cb4 100644 --- a/site2/website/versioned_docs/version-2.6.3/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.6.3/concepts-clients.md @@ -5,7 +5,7 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. @@ -38,7 +38,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and will not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This will cause readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with have a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and will not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This will cause readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with have a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is just to allow users to know how behind the reader is and is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.6.3/concepts-messaging.md b/site2/website/versioned_docs/version-2.6.3/concepts-messaging.md index 29cebdfbaedf8..d8104060d987a 100644 --- a/site2/website/versioned_docs/version-2.6.3/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.6.3/concepts-messaging.md @@ -15,8 +15,8 @@ Messages are the basic "unit" of Pulsar. The following table lists the component Component | Description :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer who produces the message. If you do not specify a producer name, the default name is used. Sequence ID | Each Pulsar message belongs to an ordered sequence on its topic. The sequence ID of the message is its order in that sequence. @@ -24,7 +24,7 @@ Publish time | The timestamp of when the message is published. The timestamp is Event time | An optional timestamp attached to a message by applications. For example, applications attach a timestamp on when the message is processed. If nothing is set to event time, the value is `0`. TypedMessageBuilder | It is used to construct a message. You can set message properties such as the message key, message value with `TypedMessageBuilder`.
    When you set `TypedMessageBuilder`, set the key as a string. If you set the key as other types, for example, an AVRO object, the key is sent as bytes, and it is difficult to get the AVRO object back on the consumer. -> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol). +> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -101,7 +101,7 @@ Messages are received from [brokers](reference-terminology.md#broker) either syn ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -378,7 +378,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -390,7 +390,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -428,7 +428,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -483,7 +483,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -506,12 +506,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication.md). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.6.3/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.6.3/concepts-multi-tenancy.md index 685c3031d4aad..8cba09547d2d3 100644 --- a/site2/website/versioned_docs/version-2.6.3/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.6.3/concepts-multi-tenancy.md @@ -5,7 +5,7 @@ sidebar_label: "Multi Tenancy" original_id: concepts-multi-tenancy --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -21,7 +21,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -29,7 +29,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.6.3/concepts-overview.md b/site2/website/versioned_docs/version-2.6.3/concepts-overview.md index c1262b38daf16..e8a2f4b9d321a 100644 --- a/site2/website/versioned_docs/version-2.6.3/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.6.3/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.6.3/concepts-replication.md b/site2/website/versioned_docs/version-2.6.3/concepts-replication.md index 6e23962e99771..799f0eb4d92c6 100644 --- a/site2/website/versioned_docs/version-2.6.3/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.6.3/concepts-replication.md @@ -5,5 +5,5 @@ sidebar_label: "Geo Replication" original_id: concepts-replication --- -Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that. +Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo.md) in Pulsar enables you to do that. diff --git a/site2/website/versioned_docs/version-2.6.3/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.6.3/concepts-tiered-storage.md index 0b45b0a2ed501..f6988e53a8cd4 100644 --- a/site2/website/versioned_docs/version-2.6.3/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.6.3/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.6.3/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.6.3/concepts-topic-compaction.md index e402c965384fd..34b7ed7fbbd31 100644 --- a/site2/website/versioned_docs/version-2.6.3/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.6.3/concepts-topic-compaction.md @@ -7,7 +7,7 @@ original_id: concepts-topic-compaction Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -24,7 +24,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.6.3/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.6.3/cookbooks-compaction.md index f436568e393de..864fb75255bfa 100644 --- a/site2/website/versioned_docs/version-2.6.3/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.6.3/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.6.3/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.6.3/cookbooks-non-persistent.md index 8eb1307c3eae5..f831119681634 100644 --- a/site2/website/versioned_docs/version-2.6.3/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.6.3/cookbooks-non-persistent.md @@ -41,7 +41,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-non-persistent-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-non-persistent-topics.md) guide. ## Enabling @@ -55,7 +55,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.6.3/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.6.3/cookbooks-partitioned.md index 4649d84057c84..cdf3bf3d14005 100644 --- a/site2/website/versioned_docs/version-2.6.3/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.6.3/cookbooks-partitioned.md @@ -7,7 +7,7 @@ original_id: cookbooks-partitioned By default, Pulsar topics are served by a single broker. Using only a single broker limits a topic's maximum throughput. *Partitioned topics* are a special type of topic that can span multiple brokers and thus allow for much higher throughput. For an explanation of how partitioned topics work, see the [Partitioned Topics](concepts-messaging.md#partitioned-topics) concepts. -You can publish to partitioned topics using Pulsar client libraries and you can [create and manage](#managing-partitioned-topics) partitioned topics using Pulsar [admin API](admin-api-overview). +You can publish to partitioned topics using Pulsar client libraries and you can [create and manage](#managing-partitioned-topics) partitioned topics using Pulsar [admin API](admin-api-overview.md). ## Publish to partitioned topics @@ -100,4 +100,4 @@ If a message has a key, it supersedes the round robin routing policy. The follow ## Manage partitioned topics -You can use Pulsar [admin API](admin-api-overview.md) to create and manage [partitioned topics](admin-api-partitioned-topics). \ No newline at end of file +You can use Pulsar [admin API](admin-api-overview.md) to create and manage [partitioned topics](admin-api-partitioned-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.6.3/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.6.3/cookbooks-retention-expiry.md index 83f69f446ce05..45198cb6aab7c 100644 --- a/site2/website/versioned_docs/version-2.6.3/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.6.3/cookbooks-retention-expiry.md @@ -16,7 +16,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.6.3/deploy-aws.md b/site2/website/versioned_docs/version-2.6.3/deploy-aws.md index 542f40b1187ea..0ac377375f585 100644 --- a/site2/website/versioned_docs/version-2.6.3/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.6.3/deploy-aws.md @@ -5,7 +5,7 @@ sidebar_label: "Amazon Web Services" original_id: deploy-aws --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. diff --git a/site2/website/versioned_docs/version-2.6.3/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.6.3/deploy-bare-metal-multi-cluster.md index bdf8dfa251b3c..8d110c5d935bb 100644 --- a/site2/website/versioned_docs/version-2.6.3/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.6.3/deploy-bare-metal-multi-cluster.md @@ -9,17 +9,17 @@ original_id: deploy-bare-metal-multi-cluster 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using it in a startup or on a single team, you had better opt for a single cluster. For instructions on deploying a single cluster, -see the guide [here](deploy-bare-metal). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo). Deploying a multi-cluster Pulsar instance involves the following basic steps: +A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md). Deploying a multi-cluster Pulsar instance involves the following basic steps: * Deploying two separate [ZooKeeper](#deploy-zookeeper) quorums: a [local](#deploy-local-zookeeper) quorum for each cluster in the instance and a [configuration store](#configuration-store) quorum for instance-wide tasks * Initializing [cluster metadata](#cluster-metadata-initialization) for each cluster @@ -29,7 +29,7 @@ A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You If you want to deploy a single Pulsar cluster, see [Clusters and Brokers](getting-started-standalone.md#start-the-cluster). > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). ## System requirement Currently, Pulsar is available for 64-bit **macOS**, **Linux**, and **Windows**. To use Pulsar, you need to install 64-bit JRE/JDK 8 or later versions. @@ -67,7 +67,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -245,7 +245,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -364,17 +364,17 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). -You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > #### Service discovery already provided by many scheduling systems -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ### Service discovery setup -The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol) of Pulsar. +The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol.md) of Pulsar. To get started setting up the built-in service of discovery of Pulsar, you need to change a few parameters in the [`conf/discovery.conf`](reference-configuration.md#service-discovery) configuration file. Set the [`zookeeperServers`](reference-configuration.md#service-discovery-zookeeperServers) parameter to the ZooKeeper quorum connection string of the cluster and the [`configurationStoreServers`](reference-configuration.md#service-discovery-configurationStoreServers) setting to the [configuration store](reference-terminology.md#configuration-store) quorum connection string. @@ -399,7 +399,7 @@ $ bin/pulsar-daemon start discovery ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.6.3/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.6.3/deploy-bare-metal.md index 2c341748b5605..5895cfe566bad 100644 --- a/site2/website/versioned_docs/version-2.6.3/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.6.3/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using Pulsar in a startup or on a single team, it is simplest to opt for a single cluster. If you do need to run a multi-cluster Pulsar instance, -see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -101,7 +101,7 @@ The extracted directory contains the following subdirectories: Directory | Contains :---------|:-------- -`bin` |[command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` |[command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `data` | The data storage directory that ZooKeeper and BookKeeper use `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -263,9 +263,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (you had better not use a different port). -`--web-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). +`--web-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should not use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (you had better not use a different port). -`--broker-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). +`--broker-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). > If you do not have a DNS server, you can use multi-host format in the service URL with the following settings: @@ -389,7 +389,7 @@ webServicePortTls=8443 ### Enable Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -407,7 +407,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker). +If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker.md). ### Start Brokers diff --git a/site2/website/versioned_docs/version-2.6.3/deploy-dcos.md b/site2/website/versioned_docs/version-2.6.3/deploy-dcos.md index f5f8d1faa7dbe..952d5f47e30fa 100644 --- a/site2/website/versioned_docs/version-2.6.3/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.6.3/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -If you want to enable all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of +If you want to enable all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.6.3/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.6.3/deploy-kubernetes.md index dc7123d000467..1aefc6ad79f71 100644 --- a/site2/website/versioned_docs/version-2.6.3/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.6.3/deploy-kubernetes.md @@ -6,6 +6,6 @@ original_id: deploy-kubernetes --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.6.3/deploy-monitoring.md b/site2/website/versioned_docs/version-2.6.3/deploy-monitoring.md index 2efc328097c00..40e918d9434fd 100644 --- a/site2/website/versioned_docs/version-2.6.3/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.6.3/deploy-monitoring.md @@ -73,7 +73,7 @@ The default port for bookie is `8000` (instead of `8080`). You can change the po You can use Prometheus to collect and store the metrics data. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes that needs to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is automatically setup with the [provided](deploy-kubernetes) instructions. +When you run Pulsar on bare metal, you can provide the list of nodes that needs to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is automatically setup with the [provided](deploy-kubernetes.md) instructions. ## Dashboards @@ -83,7 +83,7 @@ For that reason you only need to collect time series of metrics aggregated at th ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Dashboard](administration-dashboard). +The per-topic dashboard instructions are available at [Dashboard](administration-dashboard.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.6.3/develop-schema.md b/site2/website/versioned_docs/version-2.6.3/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.6.3/develop-schema.md +++ b/site2/website/versioned_docs/version-2.6.3/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.6.3/functions-deploy.md b/site2/website/versioned_docs/version-2.6.3/functions-deploy.md index 51f1140693dbe..d9496b3ed5b48 100644 --- a/site2/website/versioned_docs/version-2.6.3/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.6.3/functions-deploy.md @@ -9,12 +9,12 @@ original_id: functions-deploy To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface diff --git a/site2/website/versioned_docs/version-2.6.3/functions-develop.md b/site2/website/versioned_docs/version-2.6.3/functions-develop.md index dce77b52bb914..ca885cfdc2b04 100644 --- a/site2/website/versioned_docs/version-2.6.3/functions-develop.md +++ b/site2/website/versioned_docs/version-2.6.3/functions-develop.md @@ -852,7 +852,7 @@ public class MetricRecorderFunction implements Function { ``` -> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring) guide. +> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring.md) guide.     @@ -883,11 +883,11 @@ Currently, the feature is not available in Go. ```` ### Access metrics -To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring) in Pulsar. +To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring.md) in Pulsar. ## Security -If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker). For more details, refer to [Security settings](functions-worker.md#security-settings). +If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings). Pulsar Functions can support the following providers: diff --git a/site2/website/versioned_docs/version-2.6.3/functions-overview.md b/site2/website/versioned_docs/version-2.6.3/functions-overview.md index 25dc6029acf70..816d301e0fd0e 100644 --- a/site2/website/versioned_docs/version-2.6.3/functions-overview.md +++ b/site2/website/versioned_docs/version-2.6.3/functions-overview.md @@ -164,7 +164,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.6.3/functions-worker.md b/site2/website/versioned_docs/version-2.6.3/functions-worker.md index e6a448955b4bf..2b8968a467648 100644 --- a/site2/website/versioned_docs/version-2.6.3/functions-worker.md +++ b/site2/website/versioned_docs/version-2.6.3/functions-worker.md @@ -4,7 +4,7 @@ title: Deploy and manage functions worker sidebar_label: "Setup: Pulsar Functions Worker" original_id: functions-worker --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -178,7 +178,7 @@ brokerClientTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -198,7 +198,7 @@ authenticationProviders: [ provider1, provider2 ] ``` For *TLS Authentication* provider, follow the example below to add the necessary settings. -See [TLS Authentication](security-tls-authentication) for more details. +See [TLS Authentication](security-tls-authentication.md) for more details. ``` @@ -222,7 +222,7 @@ properties: ``` For *Token Authentication* provider, add necessary settings for `properties` if needed. -See [Token Authentication](security-jwt) for more details. +See [Token Authentication](security-jwt.md) for more details. ``` diff --git a/site2/website/versioned_docs/version-2.6.3/getting-started-docker.md b/site2/website/versioned_docs/version-2.6.3/getting-started-docker.md index b9f12f0b52688..4f20971d75330 100644 --- a/site2/website/versioned_docs/version-2.6.3/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.6.3/getting-started-docker.md @@ -50,8 +50,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -110,7 +110,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.6.3/getting-started-helm.md b/site2/website/versioned_docs/version-2.6.3/getting-started-helm.md index cb4c6945f2022..d03266c1e814e 100644 --- a/site2/website/versioned_docs/version-2.6.3/getting-started-helm.md +++ b/site2/website/versioned_docs/version-2.6.3/getting-started-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -383,7 +383,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.6.3/getting-started-standalone.md b/site2/website/versioned_docs/version-2.6.3/getting-started-standalone.md index dad19d422f359..0921bb374b98b 100644 --- a/site2/website/versioned_docs/version-2.6.3/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.6.3/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -59,7 +59,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.6.3/helm-install.md b/site2/website/versioned_docs/version-2.6.3/helm-install.md index cb72f140967e9..4559b145d09a5 100644 --- a/site2/website/versioned_docs/version-2.6.3/helm-install.md +++ b/site2/website/versioned_docs/version-2.6.3/helm-install.md @@ -21,7 +21,7 @@ Before deploying Pulsar, you need to prepare your environment. ### Tools -Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools) on your computer. +Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools.md) on your computer. ## Cloud cluster preparation @@ -37,8 +37,8 @@ To create and connect to the Kubernetes cluster, follow the instructions: ## Pulsar deployment -Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy). +Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy.md). ## Pulsar upgrade -To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade). +To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade.md). diff --git a/site2/website/versioned_docs/version-2.6.3/helm-overview.md b/site2/website/versioned_docs/version-2.6.3/helm-overview.md index a77499e81ae7a..eb7b8d2ca91c7 100644 --- a/site2/website/versioned_docs/version-2.6.3/helm-overview.md +++ b/site2/website/versioned_docs/version-2.6.3/helm-overview.md @@ -53,9 +53,9 @@ It includes support for: ## Pulsar Helm chart quick start -To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install). +This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install.md). ## Troubleshooting @@ -65,7 +65,7 @@ We have done our best to make these charts as seamless as possible. Occasionally The Apache Pulsar Helm chart contains all required dependencies. -If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm) for your first iteration. +If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm.md) for your first iteration. 1. [Preparation](helm-prepare.md) 2. [Deployment](helm-deploy.md) @@ -83,7 +83,7 @@ helm upgrade pulsar -f pulsar.yaml ``` -For more detailed information, see [Upgrading](helm-upgrade). +For more detailed information, see [Upgrading](helm-upgrade.md). ## Uninstallation diff --git a/site2/website/versioned_docs/version-2.6.3/helm-prepare.md b/site2/website/versioned_docs/version-2.6.3/helm-prepare.md index 95d7e668f9eca..5e9f2f9ef4f68 100644 --- a/site2/website/versioned_docs/version-2.6.3/helm-prepare.md +++ b/site2/website/versioned_docs/version-2.6.3/helm-prepare.md @@ -89,4 +89,4 @@ PROJECT= USE_LOCAL_SSD=true LOCAL_SSD_COUNT= ## Next Steps -Continue with the [installation of the chart](helm-deploy) once you have the cluster up and running. +Continue with the [installation of the chart](helm-deploy.md) once you have the cluster up and running. diff --git a/site2/website/versioned_docs/version-2.6.3/helm-tools.md b/site2/website/versioned_docs/version-2.6.3/helm-tools.md index efd61eae32506..6ba89006913b6 100644 --- a/site2/website/versioned_docs/version-2.6.3/helm-tools.md +++ b/site2/website/versioned_docs/version-2.6.3/helm-tools.md @@ -25,7 +25,7 @@ You can get Helm from the project's [releases page](https://github.com/helm/helm ### Next steps -Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare). +Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare.md). ## Additional information diff --git a/site2/website/versioned_docs/version-2.6.3/helm-upgrade.md b/site2/website/versioned_docs/version-2.6.3/helm-upgrade.md index 3ae1cdf2ba2fd..18838f363fb5f 100644 --- a/site2/website/versioned_docs/version-2.6.3/helm-upgrade.md +++ b/site2/website/versioned_docs/version-2.6.3/helm-upgrade.md @@ -20,7 +20,7 @@ You can retrieve your previous `--set` arguments cleanly, with `helm get values To upgrade Apache Pulsar to a newer version, follow these steps: 1. Check the change log for the specific version you would like to upgrade to. -2. Go through [deployment documentation](helm-deploy) step by step. +2. Go through [deployment documentation](helm-deploy.md) step by step. 3. Extract your previous `--set` arguments with the following command. ```bash diff --git a/site2/website/versioned_docs/version-2.6.3/io-cdc.md b/site2/website/versioned_docs/version-2.6.3/io-cdc.md index 20f16ae0d6904..e6e662884826d 100644 --- a/site2/website/versioned_docs/version-2.6.3/io-cdc.md +++ b/site2/website/versioned_docs/version-2.6.3/io-cdc.md @@ -13,8 +13,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.6.3/io-develop.md b/site2/website/versioned_docs/version-2.6.3/io-develop.md index 4a815b3b9d6f6..7a356662560c9 100644 --- a/site2/website/versioned_docs/version-2.6.3/io-develop.md +++ b/site2/website/versioned_docs/version-2.6.3/io-develop.md @@ -8,15 +8,15 @@ original_id: io-develop This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -163,7 +163,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -192,7 +192,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). diff --git a/site2/website/versioned_docs/version-2.6.3/io-overview.md b/site2/website/versioned_docs/version-2.6.3/io-overview.md index 68df2472d6334..b538604edc46f 100644 --- a/site2/website/versioned_docs/version-2.6.3/io-overview.md +++ b/site2/website/versioned_docs/version-2.6.3/io-overview.md @@ -158,7 +158,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.6.3/io-quickstart.md b/site2/website/versioned_docs/version-2.6.3/io-quickstart.md index f32e2ec8e6d15..e08e598c308f2 100644 --- a/site2/website/versioned_docs/version-2.6.3/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.6.3/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you are able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you are able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However, all the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -109,7 +109,7 @@ This section demonstrates how to connect Pulsar to Cassandra. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink.md). ::: @@ -236,11 +236,11 @@ You can create a configuration file through one of the following methods. ``` -For more information, see [Cassandra sink connector](io-cassandra-sink). +For more information, see [Cassandra sink connector](io-cassandra-sink.md). ### Create a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on them. Run the following command to create a Cassandra sink connector with sink type _cassandra_ and the config file _examples/cassandra-sink.yml_ created previously. @@ -264,7 +264,7 @@ as a Pulsar Function and writes the messages produced in the topic _test_cassand ### Inspect a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * Get the information of a Cassandra sink. @@ -428,7 +428,7 @@ to monitor a connector and perform other operations on it. ### Delete a Cassandra Sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. ```bash @@ -451,7 +451,7 @@ This section demonstrates how to connect Pulsar to PostgreSQL. ::: ->For more information, see [JDBC sink connector](io-jdbc-sink). +>For more information, see [JDBC sink connector](io-jdbc-sink.md). ### Setup a PostgreSQL cluster @@ -611,7 +611,7 @@ In this section, you need to configure a JDBC sink connector. ### Create a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on it. This example creates a sink connector and specifies the desired information. @@ -657,7 +657,7 @@ The sink has been created successfully if the following message appears. ### Inspect a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * List all running JDBC sink(s). @@ -777,7 +777,7 @@ to monitor a connector and perform other operations on it. ### Stop a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to stop a connector and perform other operations on it. ```bash @@ -805,7 +805,7 @@ The sink instance has been stopped successfully if the following message disappe ### Restart a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to restart a connector and perform other operations on it. ```bash @@ -841,7 +841,7 @@ Note that `pulsar-admin sinks localrun options` **runs a sink connector locally* ### Update a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to update a connector and perform other operations on it. This example updates the parallelism of the _pulsar-postgres-jdbc-sink_ sink connector to 2. @@ -909,7 +909,7 @@ The result shows that the parallelism is 2. ### Delete a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. This example deletes the _pulsar-postgres-jdbc-sink_ sink connector. diff --git a/site2/website/versioned_docs/version-2.6.3/io-use.md b/site2/website/versioned_docs/version-2.6.3/io-use.md index 0992b7c918b99..058d7d366f484 100644 --- a/site2/website/versioned_docs/version-2.6.3/io-use.md +++ b/site2/website/versioned_docs/version-2.6.3/io-use.md @@ -15,7 +15,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.6.3/kubernetes-helm.md b/site2/website/versioned_docs/version-2.6.3/kubernetes-helm.md index a9f768e6a96b3..ea92a0968cd7d 100644 --- a/site2/website/versioned_docs/version-2.6.3/kubernetes-helm.md +++ b/site2/website/versioned_docs/version-2.6.3/kubernetes-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.6.3/pulsar-2.0.md b/site2/website/versioned_docs/version-2.6.3/pulsar-2.0.md index 11c5e66cb62c2..752590f57b558 100644 --- a/site2/website/versioned_docs/version-2.6.3/pulsar-2.0.md +++ b/site2/website/versioned_docs/version-2.6.3/pulsar-2.0.md @@ -5,13 +5,13 @@ sidebar_label: "Pulsar 2.0" original_id: pulsar-2.0 --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.6.3/reference-cli-tools.md b/site2/website/versioned_docs/version-2.6.3/reference-cli-tools.md index a628c637adec3..73cd7580a55bb 100644 --- a/site2/website/versioned_docs/version-2.6.3/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.6.3/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.6.3/schema-manage.md b/site2/website/versioned_docs/version-2.6.3/schema-manage.md index 662b8b523726d..84d0ccdc23164 100644 --- a/site2/website/versioned_docs/version-2.6.3/schema-manage.md +++ b/site2/website/versioned_docs/version-2.6.3/schema-manage.md @@ -160,7 +160,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.6.3/schema-understand.md b/site2/website/versioned_docs/version-2.6.3/schema-understand.md index fda3d6dfecb4c..2b5fd1f44b360 100644 --- a/site2/website/versioned_docs/version-2.6.3/schema-understand.md +++ b/site2/website/versioned_docs/version-2.6.3/schema-understand.md @@ -372,7 +372,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```java diff --git a/site2/website/versioned_docs/version-2.6.3/security-athenz.md b/site2/website/versioned_docs/version-2.6.3/security-athenz.md index 947c3f470be46..8a39fe25316d0 100644 --- a/site2/website/versioned_docs/version-2.6.3/security-athenz.md +++ b/site2/website/versioned_docs/version-2.6.3/security-athenz.md @@ -76,7 +76,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.6.3/security-authorization.md b/site2/website/versioned_docs/version-2.6.3/security-authorization.md index 28200ad0744b9..ef78adc6cad0c 100644 --- a/site2/website/versioned_docs/version-2.6.3/security-authorization.md +++ b/site2/website/versioned_docs/version-2.6.3/security-authorization.md @@ -27,7 +27,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you can not only use superuser roles for administrators and clients but also for broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you can not only use superuser roles for administrators and clients but also for broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. The broker still checks the authorization of the request when the broker receives the forwarded request. @@ -58,7 +58,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -86,7 +86,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication diff --git a/site2/website/versioned_docs/version-2.6.3/security-bouncy-castle.md b/site2/website/versioned_docs/version-2.6.3/security-bouncy-castle.md index d012a7790a8ab..e8729a1188952 100644 --- a/site2/website/versioned_docs/version-2.6.3/security-bouncy-castle.md +++ b/site2/website/versioned_docs/version-2.6.3/security-bouncy-castle.md @@ -16,7 +16,7 @@ In Pulsar, security and crypto have dependencies on BouncyCastle Jars. For the d `Bouncy Castle` provides both [FIPS](https://www.bouncycastle.org/fips_faq.html) and non-FIPS version. But in a JVM, you can not include both of the 2 versions, and you need to exclude the current version before include the other. -In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. +In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption.md). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. ## Include dependencies of BC-non-FIPS diff --git a/site2/website/versioned_docs/version-2.6.3/security-extending.md b/site2/website/versioned_docs/version-2.6.3/security-extending.md index 57b4c6ad928ee..37930bb286576 100644 --- a/site2/website/versioned_docs/version-2.6.3/security-extending.md +++ b/site2/website/versioned_docs/version-2.6.3/security-extending.md @@ -9,7 +9,7 @@ Pulsar provides a way to use custom authentication and authorization mechanisms. ## Authentication -Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview). +Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview.md). You can choose to use a custom authentication mechanism by providing the implementation in the form of two plugins. One plugin is for the Client library and the other plugin is for the Pulsar Broker to validate the credentials. diff --git a/site2/website/versioned_docs/version-2.6.3/security-jwt.md b/site2/website/versioned_docs/version-2.6.3/security-jwt.md index 656ab362f71b0..91117bedff753 100644 --- a/site2/website/versioned_docs/version-2.6.3/security-jwt.md +++ b/site2/website/versioned_docs/version-2.6.3/security-jwt.md @@ -32,11 +32,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -303,7 +303,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.6.3/security-kerberos.md b/site2/website/versioned_docs/version-2.6.3/security-kerberos.md index 670586a9d1691..c49fa3bea1fce 100644 --- a/site2/website/versioned_docs/version-2.6.3/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.6.3/security-kerberos.md @@ -373,7 +373,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.6.3/security-overview.md b/site2/website/versioned_docs/version-2.6.3/security-overview.md index c7823104d145b..a1664fcb8b450 100644 --- a/site2/website/versioned_docs/version-2.6.3/security-overview.md +++ b/site2/website/versioned_docs/version-2.6.3/security-overview.md @@ -21,7 +21,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.6.3/security-tls-authentication.md b/site2/website/versioned_docs/version-2.6.3/security-tls-authentication.md index 43decf101be28..5dd2948e6cbb2 100644 --- a/site2/website/versioned_docs/version-2.6.3/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.6.3/security-tls-authentication.md @@ -7,9 +7,9 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -101,7 +101,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -121,7 +121,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.6.3/security-tls-keystore.md b/site2/website/versioned_docs/version-2.6.3/security-tls-keystore.md index 062d903e0e2ca..cb14afd1e4080 100644 --- a/site2/website/versioned_docs/version-2.6.3/security-tls-keystore.md +++ b/site2/website/versioned_docs/version-2.6.3/security-tls-keystore.md @@ -7,7 +7,7 @@ original_id: security-tls-keystore ## Overview -Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication) between clients and Apache Pulsar service. +Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication.md) between clients and Apache Pulsar service. By default it uses PEM format file configuration. This page tries to describe use [KeyStore](https://en.wikipedia.org/wiki/Java_KeyStore) type configure for TLS. @@ -165,7 +165,7 @@ This is similar to [TLS encryption configuing for client with PEM type](security For a a minimal configuration, user need to provide the TrustStore information. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties @@ -258,7 +258,7 @@ webSocketServiceEnabled=false Besides the TLS encryption configuring. The main work is configuring the KeyStore, which contains a valid CN as client role, for client. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties diff --git a/site2/website/versioned_docs/version-2.6.3/security-tls-transport.md b/site2/website/versioned_docs/version-2.6.3/security-tls-transport.md index 9638b60eab1de..29fa49120c71e 100644 --- a/site2/website/versioned_docs/version-2.6.3/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.6.3/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,9 +27,9 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). -`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ## Create TLS certificates @@ -130,7 +130,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Broker Configuration -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -201,7 +201,7 @@ The examples below show hostname verification being disabled for the Java client ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.6.3/security-token-admin.md b/site2/website/versioned_docs/version-2.6.3/security-token-admin.md index 10e91f9d30f4a..db5bd5d344492 100644 --- a/site2/website/versioned_docs/version-2.6.3/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.6.3/security-token-admin.md @@ -163,7 +163,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.6.3/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.6.3/sql-deployment-configurations.md index 28f2a3e7cb8a6..99215e81d021e 100644 --- a/site2/website/versioned_docs/version-2.6.3/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.6.3/sql-deployment-configurations.md @@ -54,7 +54,7 @@ Since Pulsar SQL is powered by [Presto](https://prestodb.io), the configuration :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.6.3/sql-overview.md b/site2/website/versioned_docs/version-2.6.3/sql-overview.md index 2bb78260265b2..8abb23ea2acbb 100644 --- a/site2/website/versioned_docs/version-2.6.3/sql-overview.md +++ b/site2/website/versioned_docs/version-2.6.3/sql-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Overview" original_id: sql-overview --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.6.3/standalone-docker.md b/site2/website/versioned_docs/version-2.6.3/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.6.3/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.6.3/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.6.3/standalone.md b/site2/website/versioned_docs/version-2.6.3/standalone.md index 220af52ce6861..e91f94140a4ac 100644 --- a/site2/website/versioned_docs/version-2.6.3/standalone.md +++ b/site2/website/versioned_docs/version-2.6.3/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -59,7 +59,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.6.4/admin-api-brokers.md b/site2/website/versioned_docs/version-2.6.4/admin-api-brokers.md index dad66178e411a..4bc9d45d3480e 100644 --- a/site2/website/versioned_docs/version-2.6.4/admin-api-brokers.md +++ b/site2/website/versioned_docs/version-2.6.4/admin-api-brokers.md @@ -12,7 +12,7 @@ Pulsar brokers consist of two components: [Brokers](reference-terminology.md#broker) can be managed via: -* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/brokers` endpoint of the admin {@inject: rest:REST:/} API * The `brokers` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin.html} object in the [Java API](client-libraries-java.md) @@ -95,7 +95,7 @@ One way to configure a Pulsar [broker](reference-terminology.md#broker) is to su But since all broker configuration in Pulsar is stored in ZooKeeper, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values. -* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. +* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. * In the Pulsar admin {@inject: rest:REST:/} API, dynamic configuration is managed through the `/admin/v2/brokers/configuration` endpoint. ### Update dynamic configuration diff --git a/site2/website/versioned_docs/version-2.6.4/admin-api-clusters.md b/site2/website/versioned_docs/version-2.6.4/admin-api-clusters.md index efd50a4c4b0c4..ac489a949d0f2 100644 --- a/site2/website/versioned_docs/version-2.6.4/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.6.4/admin-api-clusters.md @@ -10,7 +10,7 @@ servers (aka [bookies](reference-terminology.md#bookie)), and a [ZooKeeper](http Clusters can be managed via: -* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/clusters` endpoint of the admin {@inject: rest:REST:/} API * The `clusters` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md) @@ -62,7 +62,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > #### No cluster metadata initialization through the REST API or the Java admin API > @@ -86,11 +86,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. #### pulsar-admin diff --git a/site2/website/versioned_docs/version-2.6.4/admin-api-functions.md b/site2/website/versioned_docs/version-2.6.4/admin-api-functions.md index 96b7dc7ea52cd..315943492c214 100644 --- a/site2/website/versioned_docs/version-2.6.4/admin-api-functions.md +++ b/site2/website/versioned_docs/version-2.6.4/admin-api-functions.md @@ -15,9 +15,9 @@ Functions can be managed via the following methods. Method | Description ---|--- -**Admin CLI** | The [`functions`](reference-pulsar-admin.md#functions) command of the [`pulsar-admin`](reference-pulsar-admin) tool. +**Admin CLI** | The [`functions`](reference-pulsar-admin.md#functions) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool. **REST API** |The `/admin/v3/functions` endpoint of the admin {@inject: rest:REST:/} API. -**Java Admin API**| The `functions` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java). +**Java Admin API**| The `functions` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md). ## Function resources diff --git a/site2/website/versioned_docs/version-2.6.4/admin-api-namespaces.md b/site2/website/versioned_docs/version-2.6.4/admin-api-namespaces.md index 910f0e90b32d3..30e1ad3b6e1fc 100644 --- a/site2/website/versioned_docs/version-2.6.4/admin-api-namespaces.md +++ b/site2/website/versioned_docs/version-2.6.4/admin-api-namespaces.md @@ -9,7 +9,7 @@ Pulsar [namespaces](reference-terminology.md#namespace) are logical groupings of Namespaces can be managed via: -* The [`namespaces`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`namespaces`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/namespaces` endpoint of the admin {@inject: rest:REST:/} API * The `namespaces` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md) diff --git a/site2/website/versioned_docs/version-2.6.4/admin-api-non-partitioned-topics.md b/site2/website/versioned_docs/version-2.6.4/admin-api-non-partitioned-topics.md index 9cd019d2abe8f..d34d8def6473f 100644 --- a/site2/website/versioned_docs/version-2.6.4/admin-api-non-partitioned-topics.md +++ b/site2/website/versioned_docs/version-2.6.4/admin-api-non-partitioned-topics.md @@ -6,7 +6,7 @@ original_id: admin-api-non-partitioned-topics --- -You can use Pulsar's [admin API](admin-api-overview) to create and manage non-partitioned topics. +You can use Pulsar's [admin API](admin-api-overview.md) to create and manage non-partitioned topics. In all of the instructions and commands below, the topic name structure is: diff --git a/site2/website/versioned_docs/version-2.6.4/admin-api-overview.md b/site2/website/versioned_docs/version-2.6.4/admin-api-overview.md index b6186cdd0ea9a..ec19fceda0db2 100644 --- a/site2/website/versioned_docs/version-2.6.4/admin-api-overview.md +++ b/site2/website/versioned_docs/version-2.6.4/admin-api-overview.md @@ -12,7 +12,7 @@ You can currently interact with the admin interface via: - Making HTTP calls against the admin {@inject: rest:REST:/} API provided by Pulsar [brokers](reference-terminology.md#broker). For some restful apis, they might be redirected to topic owner brokers for serving with [`307 Temporary Redirect`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/307), hence the HTTP callers should handle `307 Temporary Redirect`. If you are using `curl`, you should specify `-L` to handle redirections. -- The `pulsar-admin` CLI tool, which is available in the `bin` folder of your [Pulsar installation](getting-started-standalone): +- The `pulsar-admin` CLI tool, which is available in the `bin` folder of your [Pulsar installation](getting-started-standalone.md): ```shell @@ -20,7 +20,7 @@ $ bin/pulsar-admin ``` -Full documentation for this tool can be found in the [Pulsar command-line tools](reference-pulsar-admin) doc. +Full documentation for this tool can be found in the [Pulsar command-line tools](reference-pulsar-admin.md) doc. - A Java client interface. @@ -31,11 +31,11 @@ In this document, examples from each of the three available interfaces will be s ## Admin setup -Each of Pulsar's three admin interfaces---the [`pulsar-admin`](reference-pulsar-admin) CLI tool, the [Java admin API](/api/admin), and the {@inject: rest:REST:/} API ---requires some special setup if you have [authentication](security-overview.md#authentication-providers) enabled in your Pulsar [instance](reference-terminology.md#instance). +Each of Pulsar's three admin interfaces---the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool, the [Java admin API](/api/admin), and the {@inject: rest:REST:/} API ---requires some special setup if you have [authentication](security-overview.md#authentication-providers) enabled in your Pulsar [instance](reference-terminology.md#instance). ### pulsar-admin -If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: +If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin.md) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: |Name|Description|Default| |----|-----------|-------| diff --git a/site2/website/versioned_docs/version-2.6.4/admin-api-partitioned-topics.md b/site2/website/versioned_docs/version-2.6.4/admin-api-partitioned-topics.md index eef6efdf7407b..73edab60a4039 100644 --- a/site2/website/versioned_docs/version-2.6.4/admin-api-partitioned-topics.md +++ b/site2/website/versioned_docs/version-2.6.4/admin-api-partitioned-topics.md @@ -6,7 +6,7 @@ original_id: admin-api-partitioned-topics --- -You can use Pulsar's [admin API](admin-api-overview) to create and manage partitioned topics. +You can use Pulsar's [admin API](admin-api-overview.md) to create and manage partitioned topics. In all of the instructions and commands below, the topic name structure is: diff --git a/site2/website/versioned_docs/version-2.6.4/admin-api-tenants.md b/site2/website/versioned_docs/version-2.6.4/admin-api-tenants.md index 924cbce3d41ab..27764d6da3885 100644 --- a/site2/website/versioned_docs/version-2.6.4/admin-api-tenants.md +++ b/site2/website/versioned_docs/version-2.6.4/admin-api-tenants.md @@ -5,7 +5,7 @@ sidebar_label: "Tenants" original_id: admin-api-tenants --- -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters diff --git a/site2/website/versioned_docs/version-2.6.4/administration-dashboard.md b/site2/website/versioned_docs/version-2.6.4/administration-dashboard.md index 514b076239789..25f976609b40b 100644 --- a/site2/website/versioned_docs/version-2.6.4/administration-dashboard.md +++ b/site2/website/versioned_docs/version-2.6.4/administration-dashboard.md @@ -7,7 +7,7 @@ original_id: administration-dashboard :::note -Pulsar dashboard is deprecated. If you want to manage and monitor the stats of your topics, use [Pulsar Manager](administration-pulsar-manager). +Pulsar dashboard is deprecated. If you want to manage and monitor the stats of your topics, use [Pulsar Manager](administration-pulsar-manager.md). ::: diff --git a/site2/website/versioned_docs/version-2.6.4/administration-geo.md b/site2/website/versioned_docs/version-2.6.4/administration-geo.md index 25605af3a9be5..5e545fb23a92e 100644 --- a/site2/website/versioned_docs/version-2.6.4/administration-geo.md +++ b/site2/website/versioned_docs/version-2.6.4/administration-geo.md @@ -91,7 +91,7 @@ Once you create a geo-replication namespace, any topics that producers or consum By default, messages are replicated to all clusters configured for the namespace. You can restrict replication selectively by specifying a replication list for a message, and then that message is replicated only to the subset in the replication list. -The following is an example for the [Java API](client-libraries-java). Note the use of the `setReplicationClusters` method when you construct the {@inject: javadoc:Message:/client/org/apache/pulsar/client/api/Message} object: +The following is an example for the [Java API](client-libraries-java.md). Note the use of the `setReplicationClusters` method when you construct the {@inject: javadoc:Message:/client/org/apache/pulsar/client/api/Message} object: ```java @@ -113,7 +113,7 @@ producer.newMessage() #### Topic stats -Topic-specific statistics for geo-replication topics are available via the [`pulsar-admin`](reference-pulsar-admin) tool and {@inject: rest:REST:/} API: +Topic-specific statistics for geo-replication topics are available via the [`pulsar-admin`](reference-pulsar-admin.md) tool and {@inject: rest:REST:/} API: ```shell diff --git a/site2/website/versioned_docs/version-2.6.4/administration-zk-bk.md b/site2/website/versioned_docs/version-2.6.4/administration-zk-bk.md index 4d1038a416b9e..34cb025c7541c 100644 --- a/site2/website/versioned_docs/version-2.6.4/administration-zk-bk.md +++ b/site2/website/versioned_docs/version-2.6.4/administration-zk-bk.md @@ -174,7 +174,7 @@ The [`conf/global-zookeeper.conf`](reference-configuration.md#configuration-stor BookKeeper stores all durable messages in Pulsar. BookKeeper is a distributed [write-ahead log](https://en.wikipedia.org/wiki/Write-ahead_logging) WAL system that guarantees read consistency of independent message logs calls ledgers. Individual BookKeeper servers are also called *bookies*. -> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.6.4/client-libraries-cgo.md b/site2/website/versioned_docs/version-2.6.4/client-libraries-cgo.md index c79f7bb965473..f352f942b7714 100644 --- a/site2/website/versioned_docs/version-2.6.4/client-libraries-cgo.md +++ b/site2/website/versioned_docs/version-2.6.4/client-libraries-cgo.md @@ -24,7 +24,7 @@ Currently, the following Go clients are maintained in two repositories. ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Install go package @@ -57,7 +57,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -75,7 +75,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -236,14 +236,14 @@ Parameter | Description | Default `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | `Properties`| Attach a set of application defined properties to the producer. This properties will be visible in the topic stats | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | Set the number of max pending messages across all the partitions. This setting will be used to lower the max pending messages for each partition `MaxPendingMessages(int)`, if the total exceeds the configured value.| `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | `Batching` | Control whether automatic batching of messages is enabled for the producer. | false `BatchingMaxPublishDelay` | Set the time period within which the messages sent will be batched (default: 1ms) if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or until | 1ms `BatchingMaxMessages` | Set the maximum number of messages permitted in a batch. (default: 1000) If set to a value greater than 1, messages will be queued until this threshold is reached or batch interval has elapsed | 1000 @@ -512,7 +512,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.6.4/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.6.4/client-libraries-cpp.md index bb52d9effa404..684df5fd2d825 100644 --- a/site2/website/versioned_docs/version-2.6.4/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.6.4/client-libraries-cpp.md @@ -261,7 +261,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema -This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started). +This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started.md). ### Create producer with Avro schema diff --git a/site2/website/versioned_docs/version-2.6.4/client-libraries-dotnet.md b/site2/website/versioned_docs/version-2.6.4/client-libraries-dotnet.md index c35abb42f8927..9dc1254b4782a 100644 --- a/site2/website/versioned_docs/version-2.6.4/client-libraries-dotnet.md +++ b/site2/website/versioned_docs/version-2.6.4/client-libraries-dotnet.md @@ -224,7 +224,7 @@ var client = PulsarClient.Builder() Currently, the Pulsar C# client supports the TLS (Transport Layer Security) and JWT (JSON Web Token) authentication. -If you have followed [Authentication using TLS](security-tls-authentication), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: +If you have followed [Authentication using TLS](security-tls-authentication.md), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: 1. Create an unencrypted and password-less pfx file. diff --git a/site2/website/versioned_docs/version-2.6.4/client-libraries-go.md b/site2/website/versioned_docs/version-2.6.4/client-libraries-go.md index f8b14c7e87990..1509c13332ff5 100644 --- a/site2/website/versioned_docs/version-2.6.4/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.6.4/client-libraries-go.md @@ -35,7 +35,7 @@ import "github.com/apache/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -53,7 +53,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -680,7 +680,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker @@ -700,7 +700,7 @@ opts := pulsar.ClientOptions{ ## OAuth2 authentication -To use [OAuth2 authentication](security-oauth2), you'll need to configure your client to perform the following operations. +To use [OAuth2 authentication](security-oauth2.md), you'll need to configure your client to perform the following operations. This example shows how to configure OAuth2 authentication. ```go diff --git a/site2/website/versioned_docs/version-2.6.4/client-libraries-java.md b/site2/website/versioned_docs/version-2.6.4/client-libraries-java.md index a95ea3b2c7b06..69898727edadd 100644 --- a/site2/website/versioned_docs/version-2.6.4/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.6.4/client-libraries-java.md @@ -5,7 +5,7 @@ sidebar_label: "Java" original_id: client-libraries-java --- -You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), and [reader](#reader) of a Java client are thread-safe. @@ -14,10 +14,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Includes both `pulsar-client` and `pulsar-client-admin`

    Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.

    In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -57,7 +57,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -83,7 +83,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -114,7 +114,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -237,7 +237,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -783,7 +783,7 @@ Total hash range size is 65536, so the max end of the range should be less than ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -897,7 +897,7 @@ The following schema formats are currently available for Java: ## Authentication -Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2). You can use the Pulsar Java client with all of them. +Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2.md). You can use the Pulsar Java client with all of them. ### TLS Authentication @@ -924,7 +924,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` @@ -961,7 +961,7 @@ PulsarClient client = PulsarClient.builder() ### Oauth2 -The following example shows how to use [Oauth2](security-oauth2) as an authentication provider for the Pulsar Java client. +The following example shows how to use [Oauth2](security-oauth2.md) as an authentication provider for the Pulsar Java client. You can use the factory method to configure authentication for Pulsar Java client. diff --git a/site2/website/versioned_docs/version-2.6.4/client-libraries-node.md b/site2/website/versioned_docs/version-2.6.4/client-libraries-node.md index 8d7376d3793e5..b5a138cb86528 100644 --- a/site2/website/versioned_docs/version-2.6.4/client-libraries-node.md +++ b/site2/website/versioned_docs/version-2.6.4/client-libraries-node.md @@ -44,7 +44,7 @@ Also, this library works only in Node.js 10.x or later because it uses the [`nod ::: ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here is an example for `localhost`: @@ -62,7 +62,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication), the URL will look like something like this: +If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication.md), the URL will look like something like this: ```http @@ -97,7 +97,7 @@ The following configurable parameters are available for Pulsar clients: | Parameter | Description | Default | | :-------- | :---------- | :------ | | `serviceUrl` | The connection URL for the Pulsar cluster. See [above](#connection-urls) for more info. | | -| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication) for more info. | | +| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication.md) for more info. | | | `operationTimeoutSeconds` | The timeout for Node.js client operations (creating producers, subscribing to and unsubscribing from [topics](reference-terminology.md#topic)). Retries will occur until this threshold is reached, at which point the operation will fail. | 30 | | `ioThreads` | The number of threads to use for handling connections to Pulsar [brokers](reference-terminology.md#broker). | 1 | | `messageListenerThreads` | The number of threads used by message listeners ([consumers](#consumers) and [readers](#readers)). | 1 | @@ -147,7 +147,7 @@ Pulsar Node.js producers have the following methods available: | :-------- | :---------- | :------ | | `topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer publishes messages. The topic format is `` or `//`. For example, `sample/ns1/my-topic`. | | | `producerName` | A name for the producer. If you do not explicitly assign a name, Pulsar will automatically generate a globally unique name. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | | -| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30000 | +| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30000 | | `initialSequenceId` | The initial sequence ID of the message. When producer send message, add sequence ID to message. The ID is increased each time to send. | | | `maxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `send` method will fail *unless* `blockIfQueueFull` is set to `true`. | 1000 | | `maxPendingMessagesAcrossPartitions` | The maximum size of the sum of partition's pending queue. | 50000 | diff --git a/site2/website/versioned_docs/version-2.6.4/client-libraries-python.md b/site2/website/versioned_docs/version-2.6.4/client-libraries-python.md index 459345ddb7f5b..6d4c7e6a77888 100644 --- a/site2/website/versioned_docs/version-2.6.4/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.6.4/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. diff --git a/site2/website/versioned_docs/version-2.6.4/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.6.4/client-libraries-websocket.md index 9cc5d4acb8ded..f939a32d19b13 100644 --- a/site2/website/versioned_docs/version-2.6.4/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.6.4/client-libraries-websocket.md @@ -5,14 +5,14 @@ sidebar_label: "WebSocket" original_id: client-libraries-websocket --- -Pulsar's [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API is meant to provide a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSockets you can publish and consume messages and use all the features available in the [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp) client libraries. +Pulsar's [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API is meant to provide a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSockets you can publish and consume messages and use all the features available in the [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md) client libraries. > You can use Pulsar's WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.6.4/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.6.4/concepts-architecture-overview.md index 99fdc8de886d2..7dd93f7115eee 100644 --- a/site2/website/versioned_docs/version-2.6.4/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.6.4/concepts-architecture-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Architecture" original_id: concepts-architecture-overview --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -17,20 +17,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -40,9 +40,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -130,17 +130,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -148,7 +148,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.6.4/concepts-authentication.md b/site2/website/versioned_docs/version-2.6.4/concepts-authentication.md index 5b94e89e16934..d48e25a1dff9d 100644 --- a/site2/website/versioned_docs/version-2.6.4/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.6.4/concepts-authentication.md @@ -5,5 +5,5 @@ sidebar_label: "Authentication and Authorization" original_id: concepts-authentication --- -Pulsar supports a pluggable [authentication](security-overview) mechanism which can be configured at broker and it also supports authorization to identify client and its access rights on topics and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at broker and it also supports authorization to identify client and its access rights on topics and tenants. diff --git a/site2/website/versioned_docs/version-2.6.4/concepts-clients.md b/site2/website/versioned_docs/version-2.6.4/concepts-clients.md index 88fe80b8c6381..bd5cc5dc42c43 100644 --- a/site2/website/versioned_docs/version-2.6.4/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.6.4/concepts-clients.md @@ -5,7 +5,7 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. @@ -38,7 +38,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and will not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This will cause readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with have a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and will not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This will cause readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with have a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is just to allow users to know how behind the reader is and is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.6.4/concepts-messaging.md b/site2/website/versioned_docs/version-2.6.4/concepts-messaging.md index 5ad265bdbbac6..9141911b5a426 100644 --- a/site2/website/versioned_docs/version-2.6.4/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.6.4/concepts-messaging.md @@ -15,8 +15,8 @@ Messages are the basic "unit" of Pulsar. The following table lists the component Component | Description :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer who produces the message. If you do not specify a producer name, the default name is used. Sequence ID | Each Pulsar message belongs to an ordered sequence on its topic. The sequence ID of the message is its order in that sequence. @@ -24,7 +24,7 @@ Publish time | The timestamp of when the message is published. The timestamp is Event time | An optional timestamp attached to a message by applications. For example, applications attach a timestamp on when the message is processed. If nothing is set to event time, the value is `0`. TypedMessageBuilder | It is used to construct a message. You can set message properties such as the message key, message value with `TypedMessageBuilder`.
    When you set `TypedMessageBuilder`, set the key as a string. If you set the key as other types, for example, an AVRO object, the key is sent as bytes, and it is difficult to get the AVRO object back on the consumer. -> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol). +> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -101,7 +101,7 @@ Messages are received from [brokers](reference-terminology.md#broker) either syn ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -375,7 +375,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -387,7 +387,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -425,7 +425,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -480,7 +480,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -503,12 +503,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication.md). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.6.4/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.6.4/concepts-multi-tenancy.md index 685c3031d4aad..8cba09547d2d3 100644 --- a/site2/website/versioned_docs/version-2.6.4/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.6.4/concepts-multi-tenancy.md @@ -5,7 +5,7 @@ sidebar_label: "Multi Tenancy" original_id: concepts-multi-tenancy --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -21,7 +21,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -29,7 +29,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.6.4/concepts-overview.md b/site2/website/versioned_docs/version-2.6.4/concepts-overview.md index c1262b38daf16..e8a2f4b9d321a 100644 --- a/site2/website/versioned_docs/version-2.6.4/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.6.4/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.6.4/concepts-replication.md b/site2/website/versioned_docs/version-2.6.4/concepts-replication.md index 6e23962e99771..799f0eb4d92c6 100644 --- a/site2/website/versioned_docs/version-2.6.4/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.6.4/concepts-replication.md @@ -5,5 +5,5 @@ sidebar_label: "Geo Replication" original_id: concepts-replication --- -Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that. +Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo.md) in Pulsar enables you to do that. diff --git a/site2/website/versioned_docs/version-2.6.4/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.6.4/concepts-tiered-storage.md index 0b45b0a2ed501..f6988e53a8cd4 100644 --- a/site2/website/versioned_docs/version-2.6.4/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.6.4/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.6.4/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.6.4/concepts-topic-compaction.md index e402c965384fd..34b7ed7fbbd31 100644 --- a/site2/website/versioned_docs/version-2.6.4/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.6.4/concepts-topic-compaction.md @@ -7,7 +7,7 @@ original_id: concepts-topic-compaction Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -24,7 +24,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.6.4/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.6.4/cookbooks-compaction.md index f436568e393de..864fb75255bfa 100644 --- a/site2/website/versioned_docs/version-2.6.4/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.6.4/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.6.4/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.6.4/cookbooks-non-persistent.md index 8eb1307c3eae5..f831119681634 100644 --- a/site2/website/versioned_docs/version-2.6.4/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.6.4/cookbooks-non-persistent.md @@ -41,7 +41,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-non-persistent-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-non-persistent-topics.md) guide. ## Enabling @@ -55,7 +55,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.6.4/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.6.4/cookbooks-partitioned.md index 4649d84057c84..cdf3bf3d14005 100644 --- a/site2/website/versioned_docs/version-2.6.4/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.6.4/cookbooks-partitioned.md @@ -7,7 +7,7 @@ original_id: cookbooks-partitioned By default, Pulsar topics are served by a single broker. Using only a single broker limits a topic's maximum throughput. *Partitioned topics* are a special type of topic that can span multiple brokers and thus allow for much higher throughput. For an explanation of how partitioned topics work, see the [Partitioned Topics](concepts-messaging.md#partitioned-topics) concepts. -You can publish to partitioned topics using Pulsar client libraries and you can [create and manage](#managing-partitioned-topics) partitioned topics using Pulsar [admin API](admin-api-overview). +You can publish to partitioned topics using Pulsar client libraries and you can [create and manage](#managing-partitioned-topics) partitioned topics using Pulsar [admin API](admin-api-overview.md). ## Publish to partitioned topics @@ -100,4 +100,4 @@ If a message has a key, it supersedes the round robin routing policy. The follow ## Manage partitioned topics -You can use Pulsar [admin API](admin-api-overview.md) to create and manage [partitioned topics](admin-api-partitioned-topics). \ No newline at end of file +You can use Pulsar [admin API](admin-api-overview.md) to create and manage [partitioned topics](admin-api-partitioned-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.6.4/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.6.4/cookbooks-retention-expiry.md index 83f69f446ce05..45198cb6aab7c 100644 --- a/site2/website/versioned_docs/version-2.6.4/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.6.4/cookbooks-retention-expiry.md @@ -16,7 +16,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.6.4/deploy-aws.md b/site2/website/versioned_docs/version-2.6.4/deploy-aws.md index 071effeffde24..93c389b56e2cf 100644 --- a/site2/website/versioned_docs/version-2.6.4/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.6.4/deploy-aws.md @@ -5,7 +5,7 @@ sidebar_label: "Amazon Web Services" original_id: deploy-aws --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -210,7 +210,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors) , edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md) , edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.6.4/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.6.4/deploy-bare-metal-multi-cluster.md index bdf8dfa251b3c..8d110c5d935bb 100644 --- a/site2/website/versioned_docs/version-2.6.4/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.6.4/deploy-bare-metal-multi-cluster.md @@ -9,17 +9,17 @@ original_id: deploy-bare-metal-multi-cluster 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using it in a startup or on a single team, you had better opt for a single cluster. For instructions on deploying a single cluster, -see the guide [here](deploy-bare-metal). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo). Deploying a multi-cluster Pulsar instance involves the following basic steps: +A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md). Deploying a multi-cluster Pulsar instance involves the following basic steps: * Deploying two separate [ZooKeeper](#deploy-zookeeper) quorums: a [local](#deploy-local-zookeeper) quorum for each cluster in the instance and a [configuration store](#configuration-store) quorum for instance-wide tasks * Initializing [cluster metadata](#cluster-metadata-initialization) for each cluster @@ -29,7 +29,7 @@ A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You If you want to deploy a single Pulsar cluster, see [Clusters and Brokers](getting-started-standalone.md#start-the-cluster). > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). ## System requirement Currently, Pulsar is available for 64-bit **macOS**, **Linux**, and **Windows**. To use Pulsar, you need to install 64-bit JRE/JDK 8 or later versions. @@ -67,7 +67,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -245,7 +245,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -364,17 +364,17 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). -You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > #### Service discovery already provided by many scheduling systems -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ### Service discovery setup -The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol) of Pulsar. +The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol.md) of Pulsar. To get started setting up the built-in service of discovery of Pulsar, you need to change a few parameters in the [`conf/discovery.conf`](reference-configuration.md#service-discovery) configuration file. Set the [`zookeeperServers`](reference-configuration.md#service-discovery-zookeeperServers) parameter to the ZooKeeper quorum connection string of the cluster and the [`configurationStoreServers`](reference-configuration.md#service-discovery-configurationStoreServers) setting to the [configuration store](reference-terminology.md#configuration-store) quorum connection string. @@ -399,7 +399,7 @@ $ bin/pulsar-daemon start discovery ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.6.4/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.6.4/deploy-bare-metal.md index 5b69db21ac4dc..89d31c9b2b09a 100644 --- a/site2/website/versioned_docs/version-2.6.4/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.6.4/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using Pulsar in a startup or on a single team, it is simplest to opt for a single cluster. If you do need to run a multi-cluster Pulsar instance, -see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -99,7 +99,7 @@ The extracted directory contains the following subdirectories: Directory | Contains :---------|:-------- -`bin` |[command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` |[command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `data` | The data storage directory that ZooKeeper and BookKeeper use `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -261,9 +261,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (you had better not use a different port). -`--web-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). +`--web-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should not use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (you had better not use a different port). -`--broker-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). +`--broker-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). > If you do not have a DNS server, you can use multi-host format in the service URL with the following settings: @@ -401,7 +401,7 @@ webServicePortTls=8443 ### Enable Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -419,7 +419,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker). +If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker.md). ### Start Brokers diff --git a/site2/website/versioned_docs/version-2.6.4/deploy-dcos.md b/site2/website/versioned_docs/version-2.6.4/deploy-dcos.md index f5f8d1faa7dbe..952d5f47e30fa 100644 --- a/site2/website/versioned_docs/version-2.6.4/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.6.4/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -If you want to enable all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of +If you want to enable all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.6.4/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.6.4/deploy-kubernetes.md index dc7123d000467..1aefc6ad79f71 100644 --- a/site2/website/versioned_docs/version-2.6.4/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.6.4/deploy-kubernetes.md @@ -6,6 +6,6 @@ original_id: deploy-kubernetes --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.6.4/deploy-monitoring.md b/site2/website/versioned_docs/version-2.6.4/deploy-monitoring.md index 2efc328097c00..40e918d9434fd 100644 --- a/site2/website/versioned_docs/version-2.6.4/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.6.4/deploy-monitoring.md @@ -73,7 +73,7 @@ The default port for bookie is `8000` (instead of `8080`). You can change the po You can use Prometheus to collect and store the metrics data. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes that needs to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is automatically setup with the [provided](deploy-kubernetes) instructions. +When you run Pulsar on bare metal, you can provide the list of nodes that needs to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is automatically setup with the [provided](deploy-kubernetes.md) instructions. ## Dashboards @@ -83,7 +83,7 @@ For that reason you only need to collect time series of metrics aggregated at th ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Dashboard](administration-dashboard). +The per-topic dashboard instructions are available at [Dashboard](administration-dashboard.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.6.4/develop-schema.md b/site2/website/versioned_docs/version-2.6.4/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.6.4/develop-schema.md +++ b/site2/website/versioned_docs/version-2.6.4/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.6.4/functions-deploy.md b/site2/website/versioned_docs/version-2.6.4/functions-deploy.md index 51f1140693dbe..d9496b3ed5b48 100644 --- a/site2/website/versioned_docs/version-2.6.4/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.6.4/functions-deploy.md @@ -9,12 +9,12 @@ original_id: functions-deploy To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface diff --git a/site2/website/versioned_docs/version-2.6.4/functions-develop.md b/site2/website/versioned_docs/version-2.6.4/functions-develop.md index 31c8f3a967a29..b281774108645 100644 --- a/site2/website/versioned_docs/version-2.6.4/functions-develop.md +++ b/site2/website/versioned_docs/version-2.6.4/functions-develop.md @@ -850,7 +850,7 @@ public class MetricRecorderFunction implements Function { ``` -> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring) guide. +> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring.md) guide.     @@ -881,11 +881,11 @@ Currently, the feature is not available in Go. ```` ### Access metrics -To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring) in Pulsar. +To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring.md) in Pulsar. ## Security -If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker). For more details, refer to [Security settings](functions-worker.md#security-settings). +If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings). Pulsar Functions can support the following providers: diff --git a/site2/website/versioned_docs/version-2.6.4/functions-overview.md b/site2/website/versioned_docs/version-2.6.4/functions-overview.md index 25dc6029acf70..816d301e0fd0e 100644 --- a/site2/website/versioned_docs/version-2.6.4/functions-overview.md +++ b/site2/website/versioned_docs/version-2.6.4/functions-overview.md @@ -164,7 +164,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.6.4/functions-worker.md b/site2/website/versioned_docs/version-2.6.4/functions-worker.md index df1d8d060c7e2..91b129c233a1d 100644 --- a/site2/website/versioned_docs/version-2.6.4/functions-worker.md +++ b/site2/website/versioned_docs/version-2.6.4/functions-worker.md @@ -4,7 +4,7 @@ title: Deploy and manage functions worker sidebar_label: "Setup: Pulsar Functions Worker" original_id: functions-worker --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -178,7 +178,7 @@ brokerClientTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -198,7 +198,7 @@ authenticationProviders: [ provider1, provider2 ] ``` For *TLS Authentication* provider, follow the example below to add the necessary settings. -See [TLS Authentication](security-tls-authentication) for more details. +See [TLS Authentication](security-tls-authentication.md) for more details. ``` @@ -222,7 +222,7 @@ properties: ``` For *Token Authentication* provider, add necessary settings for `properties` if needed. -See [Token Authentication](security-jwt) for more details. +See [Token Authentication](security-jwt.md) for more details. ``` diff --git a/site2/website/versioned_docs/version-2.6.4/getting-started-docker.md b/site2/website/versioned_docs/version-2.6.4/getting-started-docker.md index b9f12f0b52688..4f20971d75330 100644 --- a/site2/website/versioned_docs/version-2.6.4/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.6.4/getting-started-docker.md @@ -50,8 +50,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -110,7 +110,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.6.4/getting-started-helm.md b/site2/website/versioned_docs/version-2.6.4/getting-started-helm.md index 406276c62cd07..346cc7f675c09 100644 --- a/site2/website/versioned_docs/version-2.6.4/getting-started-helm.md +++ b/site2/website/versioned_docs/version-2.6.4/getting-started-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -386,7 +386,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.6.4/getting-started-standalone.md b/site2/website/versioned_docs/version-2.6.4/getting-started-standalone.md index dad19d422f359..0921bb374b98b 100644 --- a/site2/website/versioned_docs/version-2.6.4/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.6.4/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -59,7 +59,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.6.4/helm-install.md b/site2/website/versioned_docs/version-2.6.4/helm-install.md index edb12260f010e..1f4d5eb69d5dd 100644 --- a/site2/website/versioned_docs/version-2.6.4/helm-install.md +++ b/site2/website/versioned_docs/version-2.6.4/helm-install.md @@ -21,7 +21,7 @@ Before deploying Pulsar, you need to prepare your environment. ### Tools -Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools) on your computer. +Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools.md) on your computer. ## Cloud cluster preparation @@ -37,8 +37,8 @@ To create and connect to the Kubernetes cluster, follow the instructions: ## Pulsar deployment -Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy). +Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy.md). ## Pulsar upgrade -To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade). +To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade.md). diff --git a/site2/website/versioned_docs/version-2.6.4/helm-overview.md b/site2/website/versioned_docs/version-2.6.4/helm-overview.md index a77499e81ae7a..eb7b8d2ca91c7 100644 --- a/site2/website/versioned_docs/version-2.6.4/helm-overview.md +++ b/site2/website/versioned_docs/version-2.6.4/helm-overview.md @@ -53,9 +53,9 @@ It includes support for: ## Pulsar Helm chart quick start -To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install). +This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install.md). ## Troubleshooting @@ -65,7 +65,7 @@ We have done our best to make these charts as seamless as possible. Occasionally The Apache Pulsar Helm chart contains all required dependencies. -If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm) for your first iteration. +If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm.md) for your first iteration. 1. [Preparation](helm-prepare.md) 2. [Deployment](helm-deploy.md) @@ -83,7 +83,7 @@ helm upgrade pulsar -f pulsar.yaml ``` -For more detailed information, see [Upgrading](helm-upgrade). +For more detailed information, see [Upgrading](helm-upgrade.md). ## Uninstallation diff --git a/site2/website/versioned_docs/version-2.6.4/helm-prepare.md b/site2/website/versioned_docs/version-2.6.4/helm-prepare.md index 95d7e668f9eca..5e9f2f9ef4f68 100644 --- a/site2/website/versioned_docs/version-2.6.4/helm-prepare.md +++ b/site2/website/versioned_docs/version-2.6.4/helm-prepare.md @@ -89,4 +89,4 @@ PROJECT= USE_LOCAL_SSD=true LOCAL_SSD_COUNT= ## Next Steps -Continue with the [installation of the chart](helm-deploy) once you have the cluster up and running. +Continue with the [installation of the chart](helm-deploy.md) once you have the cluster up and running. diff --git a/site2/website/versioned_docs/version-2.6.4/helm-tools.md b/site2/website/versioned_docs/version-2.6.4/helm-tools.md index efd61eae32506..6ba89006913b6 100644 --- a/site2/website/versioned_docs/version-2.6.4/helm-tools.md +++ b/site2/website/versioned_docs/version-2.6.4/helm-tools.md @@ -25,7 +25,7 @@ You can get Helm from the project's [releases page](https://github.com/helm/helm ### Next steps -Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare). +Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare.md). ## Additional information diff --git a/site2/website/versioned_docs/version-2.6.4/helm-upgrade.md b/site2/website/versioned_docs/version-2.6.4/helm-upgrade.md index 3ae1cdf2ba2fd..18838f363fb5f 100644 --- a/site2/website/versioned_docs/version-2.6.4/helm-upgrade.md +++ b/site2/website/versioned_docs/version-2.6.4/helm-upgrade.md @@ -20,7 +20,7 @@ You can retrieve your previous `--set` arguments cleanly, with `helm get values To upgrade Apache Pulsar to a newer version, follow these steps: 1. Check the change log for the specific version you would like to upgrade to. -2. Go through [deployment documentation](helm-deploy) step by step. +2. Go through [deployment documentation](helm-deploy.md) step by step. 3. Extract your previous `--set` arguments with the following command. ```bash diff --git a/site2/website/versioned_docs/version-2.6.4/io-cdc.md b/site2/website/versioned_docs/version-2.6.4/io-cdc.md index 20f16ae0d6904..e6e662884826d 100644 --- a/site2/website/versioned_docs/version-2.6.4/io-cdc.md +++ b/site2/website/versioned_docs/version-2.6.4/io-cdc.md @@ -13,8 +13,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.6.4/io-develop.md b/site2/website/versioned_docs/version-2.6.4/io-develop.md index 4a815b3b9d6f6..7a356662560c9 100644 --- a/site2/website/versioned_docs/version-2.6.4/io-develop.md +++ b/site2/website/versioned_docs/version-2.6.4/io-develop.md @@ -8,15 +8,15 @@ original_id: io-develop This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -163,7 +163,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -192,7 +192,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). diff --git a/site2/website/versioned_docs/version-2.6.4/io-overview.md b/site2/website/versioned_docs/version-2.6.4/io-overview.md index 68df2472d6334..b538604edc46f 100644 --- a/site2/website/versioned_docs/version-2.6.4/io-overview.md +++ b/site2/website/versioned_docs/version-2.6.4/io-overview.md @@ -158,7 +158,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.6.4/io-quickstart.md b/site2/website/versioned_docs/version-2.6.4/io-quickstart.md index f32e2ec8e6d15..e08e598c308f2 100644 --- a/site2/website/versioned_docs/version-2.6.4/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.6.4/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you are able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you are able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However, all the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -109,7 +109,7 @@ This section demonstrates how to connect Pulsar to Cassandra. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink.md). ::: @@ -236,11 +236,11 @@ You can create a configuration file through one of the following methods. ``` -For more information, see [Cassandra sink connector](io-cassandra-sink). +For more information, see [Cassandra sink connector](io-cassandra-sink.md). ### Create a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on them. Run the following command to create a Cassandra sink connector with sink type _cassandra_ and the config file _examples/cassandra-sink.yml_ created previously. @@ -264,7 +264,7 @@ as a Pulsar Function and writes the messages produced in the topic _test_cassand ### Inspect a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * Get the information of a Cassandra sink. @@ -428,7 +428,7 @@ to monitor a connector and perform other operations on it. ### Delete a Cassandra Sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. ```bash @@ -451,7 +451,7 @@ This section demonstrates how to connect Pulsar to PostgreSQL. ::: ->For more information, see [JDBC sink connector](io-jdbc-sink). +>For more information, see [JDBC sink connector](io-jdbc-sink.md). ### Setup a PostgreSQL cluster @@ -611,7 +611,7 @@ In this section, you need to configure a JDBC sink connector. ### Create a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on it. This example creates a sink connector and specifies the desired information. @@ -657,7 +657,7 @@ The sink has been created successfully if the following message appears. ### Inspect a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * List all running JDBC sink(s). @@ -777,7 +777,7 @@ to monitor a connector and perform other operations on it. ### Stop a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to stop a connector and perform other operations on it. ```bash @@ -805,7 +805,7 @@ The sink instance has been stopped successfully if the following message disappe ### Restart a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to restart a connector and perform other operations on it. ```bash @@ -841,7 +841,7 @@ Note that `pulsar-admin sinks localrun options` **runs a sink connector locally* ### Update a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to update a connector and perform other operations on it. This example updates the parallelism of the _pulsar-postgres-jdbc-sink_ sink connector to 2. @@ -909,7 +909,7 @@ The result shows that the parallelism is 2. ### Delete a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. This example deletes the _pulsar-postgres-jdbc-sink_ sink connector. diff --git a/site2/website/versioned_docs/version-2.6.4/io-use.md b/site2/website/versioned_docs/version-2.6.4/io-use.md index 460aa401b6aa9..90501c5bdb93b 100644 --- a/site2/website/versioned_docs/version-2.6.4/io-use.md +++ b/site2/website/versioned_docs/version-2.6.4/io-use.md @@ -15,7 +15,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.6.4/kubernetes-helm.md b/site2/website/versioned_docs/version-2.6.4/kubernetes-helm.md index a9f768e6a96b3..ea92a0968cd7d 100644 --- a/site2/website/versioned_docs/version-2.6.4/kubernetes-helm.md +++ b/site2/website/versioned_docs/version-2.6.4/kubernetes-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.6.4/pulsar-2.0.md b/site2/website/versioned_docs/version-2.6.4/pulsar-2.0.md index 11c5e66cb62c2..752590f57b558 100644 --- a/site2/website/versioned_docs/version-2.6.4/pulsar-2.0.md +++ b/site2/website/versioned_docs/version-2.6.4/pulsar-2.0.md @@ -5,13 +5,13 @@ sidebar_label: "Pulsar 2.0" original_id: pulsar-2.0 --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.6.4/reference-cli-tools.md b/site2/website/versioned_docs/version-2.6.4/reference-cli-tools.md index a628c637adec3..73cd7580a55bb 100644 --- a/site2/website/versioned_docs/version-2.6.4/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.6.4/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.6.4/schema-manage.md b/site2/website/versioned_docs/version-2.6.4/schema-manage.md index 662b8b523726d..84d0ccdc23164 100644 --- a/site2/website/versioned_docs/version-2.6.4/schema-manage.md +++ b/site2/website/versioned_docs/version-2.6.4/schema-manage.md @@ -160,7 +160,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.6.4/schema-understand.md b/site2/website/versioned_docs/version-2.6.4/schema-understand.md index fda3d6dfecb4c..2b5fd1f44b360 100644 --- a/site2/website/versioned_docs/version-2.6.4/schema-understand.md +++ b/site2/website/versioned_docs/version-2.6.4/schema-understand.md @@ -372,7 +372,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```java diff --git a/site2/website/versioned_docs/version-2.6.4/security-athenz.md b/site2/website/versioned_docs/version-2.6.4/security-athenz.md index 947c3f470be46..8a39fe25316d0 100644 --- a/site2/website/versioned_docs/version-2.6.4/security-athenz.md +++ b/site2/website/versioned_docs/version-2.6.4/security-athenz.md @@ -76,7 +76,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.6.4/security-authorization.md b/site2/website/versioned_docs/version-2.6.4/security-authorization.md index 28200ad0744b9..ef78adc6cad0c 100644 --- a/site2/website/versioned_docs/version-2.6.4/security-authorization.md +++ b/site2/website/versioned_docs/version-2.6.4/security-authorization.md @@ -27,7 +27,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you can not only use superuser roles for administrators and clients but also for broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you can not only use superuser roles for administrators and clients but also for broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. The broker still checks the authorization of the request when the broker receives the forwarded request. @@ -58,7 +58,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -86,7 +86,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication diff --git a/site2/website/versioned_docs/version-2.6.4/security-bouncy-castle.md b/site2/website/versioned_docs/version-2.6.4/security-bouncy-castle.md index d012a7790a8ab..e8729a1188952 100644 --- a/site2/website/versioned_docs/version-2.6.4/security-bouncy-castle.md +++ b/site2/website/versioned_docs/version-2.6.4/security-bouncy-castle.md @@ -16,7 +16,7 @@ In Pulsar, security and crypto have dependencies on BouncyCastle Jars. For the d `Bouncy Castle` provides both [FIPS](https://www.bouncycastle.org/fips_faq.html) and non-FIPS version. But in a JVM, you can not include both of the 2 versions, and you need to exclude the current version before include the other. -In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. +In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption.md). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. ## Include dependencies of BC-non-FIPS diff --git a/site2/website/versioned_docs/version-2.6.4/security-extending.md b/site2/website/versioned_docs/version-2.6.4/security-extending.md index 57b4c6ad928ee..37930bb286576 100644 --- a/site2/website/versioned_docs/version-2.6.4/security-extending.md +++ b/site2/website/versioned_docs/version-2.6.4/security-extending.md @@ -9,7 +9,7 @@ Pulsar provides a way to use custom authentication and authorization mechanisms. ## Authentication -Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview). +Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview.md). You can choose to use a custom authentication mechanism by providing the implementation in the form of two plugins. One plugin is for the Client library and the other plugin is for the Pulsar Broker to validate the credentials. diff --git a/site2/website/versioned_docs/version-2.6.4/security-jwt.md b/site2/website/versioned_docs/version-2.6.4/security-jwt.md index 8f78751b81c45..3a503709dfb81 100644 --- a/site2/website/versioned_docs/version-2.6.4/security-jwt.md +++ b/site2/website/versioned_docs/version-2.6.4/security-jwt.md @@ -32,11 +32,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -291,7 +291,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.6.4/security-kerberos.md b/site2/website/versioned_docs/version-2.6.4/security-kerberos.md index 670586a9d1691..c49fa3bea1fce 100644 --- a/site2/website/versioned_docs/version-2.6.4/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.6.4/security-kerberos.md @@ -373,7 +373,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.6.4/security-overview.md b/site2/website/versioned_docs/version-2.6.4/security-overview.md index c7823104d145b..a1664fcb8b450 100644 --- a/site2/website/versioned_docs/version-2.6.4/security-overview.md +++ b/site2/website/versioned_docs/version-2.6.4/security-overview.md @@ -21,7 +21,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.6.4/security-tls-authentication.md b/site2/website/versioned_docs/version-2.6.4/security-tls-authentication.md index 20a6604a6c1fe..e1b67ffa05b78 100644 --- a/site2/website/versioned_docs/version-2.6.4/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.6.4/security-tls-authentication.md @@ -7,9 +7,9 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -101,7 +101,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -121,7 +121,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.6.4/security-tls-keystore.md b/site2/website/versioned_docs/version-2.6.4/security-tls-keystore.md index f85af6b91e1bd..f44fdfc0be5b8 100644 --- a/site2/website/versioned_docs/version-2.6.4/security-tls-keystore.md +++ b/site2/website/versioned_docs/version-2.6.4/security-tls-keystore.md @@ -7,7 +7,7 @@ original_id: security-tls-keystore ## Overview -Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication) between clients and Apache Pulsar service. +Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication.md) between clients and Apache Pulsar service. By default it uses PEM format file configuration. This page tries to describe use [KeyStore](https://en.wikipedia.org/wiki/Java_KeyStore) type configure for TLS. @@ -165,7 +165,7 @@ This is similar to [TLS encryption configuing for client with PEM type](security For a a minimal configuration, user need to provide the TrustStore information. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties @@ -258,7 +258,7 @@ webSocketServiceEnabled=false Besides the TLS encryption configuring. The main work is configuring the KeyStore, which contains a valid CN as client role, for client. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties diff --git a/site2/website/versioned_docs/version-2.6.4/security-tls-transport.md b/site2/website/versioned_docs/version-2.6.4/security-tls-transport.md index 9638b60eab1de..29fa49120c71e 100644 --- a/site2/website/versioned_docs/version-2.6.4/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.6.4/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,9 +27,9 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). -`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ## Create TLS certificates @@ -130,7 +130,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Broker Configuration -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -201,7 +201,7 @@ The examples below show hostname verification being disabled for the Java client ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.6.4/security-token-admin.md b/site2/website/versioned_docs/version-2.6.4/security-token-admin.md index 6bef23b106415..39bb6178e2bf3 100644 --- a/site2/website/versioned_docs/version-2.6.4/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.6.4/security-token-admin.md @@ -163,7 +163,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.6.4/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.6.4/sql-deployment-configurations.md index 3c7c143848156..6ba4d7c048072 100644 --- a/site2/website/versioned_docs/version-2.6.4/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.6.4/sql-deployment-configurations.md @@ -54,7 +54,7 @@ Since Pulsar SQL is powered by [Presto](https://prestosql.io), the configuration :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.6.4/sql-overview.md b/site2/website/versioned_docs/version-2.6.4/sql-overview.md index 2bb78260265b2..8abb23ea2acbb 100644 --- a/site2/website/versioned_docs/version-2.6.4/sql-overview.md +++ b/site2/website/versioned_docs/version-2.6.4/sql-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Overview" original_id: sql-overview --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.6.4/standalone-docker.md b/site2/website/versioned_docs/version-2.6.4/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.6.4/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.6.4/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.6.4/standalone.md b/site2/website/versioned_docs/version-2.6.4/standalone.md index 220af52ce6861..e91f94140a4ac 100644 --- a/site2/website/versioned_docs/version-2.6.4/standalone.md +++ b/site2/website/versioned_docs/version-2.6.4/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -59,7 +59,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.7.0/admin-api-brokers.md b/site2/website/versioned_docs/version-2.7.0/admin-api-brokers.md index 751bc14983205..61047c4d631d1 100644 --- a/site2/website/versioned_docs/version-2.7.0/admin-api-brokers.md +++ b/site2/website/versioned_docs/version-2.7.0/admin-api-brokers.md @@ -18,7 +18,7 @@ Pulsar brokers consist of two components: [Brokers](reference-terminology.md#broker) can be managed via: -* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/brokers` endpoint of the admin {@inject: rest:REST:/} API * The `brokers` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin.html} object in the [Java API](client-libraries-java.md) @@ -123,7 +123,7 @@ One way to configure a Pulsar [broker](reference-terminology.md#broker) is to su But since all broker configuration in Pulsar is stored in ZooKeeper, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values. -* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. +* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. * In the Pulsar admin {@inject: rest:REST:/} API, dynamic configuration is managed through the `/admin/v2/brokers/configuration` endpoint. ### Update dynamic configuration diff --git a/site2/website/versioned_docs/version-2.7.0/admin-api-clusters.md b/site2/website/versioned_docs/version-2.7.0/admin-api-clusters.md index 3eb238506cc69..e0e9fb5f91f65 100644 --- a/site2/website/versioned_docs/version-2.7.0/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.7.0/admin-api-clusters.md @@ -16,7 +16,7 @@ servers (aka [bookies](reference-terminology.md#bookie)), and a [ZooKeeper](http Clusters can be managed via: -* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/clusters` endpoint of the admin {@inject: rest:REST:/} API * The `clusters` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md) @@ -79,7 +79,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -103,11 +103,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block -If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: +If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin.md) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: |Name|Description|Default| |----|-----------|-------| diff --git a/site2/website/versioned_docs/version-2.7.0/admin-api-partitioned-topics.md b/site2/website/versioned_docs/version-2.7.0/admin-api-partitioned-topics.md index 6734586da1718..5ce182282e032 100644 --- a/site2/website/versioned_docs/version-2.7.0/admin-api-partitioned-topics.md +++ b/site2/website/versioned_docs/version-2.7.0/admin-api-partitioned-topics.md @@ -5,4 +5,4 @@ sidebar_label: "Partitioned topics" original_id: admin-api-partitioned-topics --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.7.0/admin-api-persistent-topics.md b/site2/website/versioned_docs/version-2.7.0/admin-api-persistent-topics.md index b6d293b1e6be5..50d135b72f542 100644 --- a/site2/website/versioned_docs/version-2.7.0/admin-api-persistent-topics.md +++ b/site2/website/versioned_docs/version-2.7.0/admin-api-persistent-topics.md @@ -5,4 +5,4 @@ sidebar_label: "Persistent topics" original_id: admin-api-persistent-topics --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.7.0/admin-api-tenants.md b/site2/website/versioned_docs/version-2.7.0/admin-api-tenants.md index 63afe58620c62..d78aa2e55f4c3 100644 --- a/site2/website/versioned_docs/version-2.7.0/admin-api-tenants.md +++ b/site2/website/versioned_docs/version-2.7.0/admin-api-tenants.md @@ -11,7 +11,7 @@ import TabItem from '@theme/TabItem'; ```` -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -108,7 +108,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.7.0/client-libraries-cgo.md b/site2/website/versioned_docs/version-2.7.0/client-libraries-cgo.md index c79f7bb965473..f352f942b7714 100644 --- a/site2/website/versioned_docs/version-2.7.0/client-libraries-cgo.md +++ b/site2/website/versioned_docs/version-2.7.0/client-libraries-cgo.md @@ -24,7 +24,7 @@ Currently, the following Go clients are maintained in two repositories. ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Install go package @@ -57,7 +57,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -75,7 +75,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -236,14 +236,14 @@ Parameter | Description | Default `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | `Properties`| Attach a set of application defined properties to the producer. This properties will be visible in the topic stats | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | Set the number of max pending messages across all the partitions. This setting will be used to lower the max pending messages for each partition `MaxPendingMessages(int)`, if the total exceeds the configured value.| `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | `Batching` | Control whether automatic batching of messages is enabled for the producer. | false `BatchingMaxPublishDelay` | Set the time period within which the messages sent will be batched (default: 1ms) if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or until | 1ms `BatchingMaxMessages` | Set the maximum number of messages permitted in a batch. (default: 1000) If set to a value greater than 1, messages will be queued until this threshold is reached or batch interval has elapsed | 1000 @@ -512,7 +512,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.7.0/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.7.0/client-libraries-cpp.md index bb52d9effa404..684df5fd2d825 100644 --- a/site2/website/versioned_docs/version-2.7.0/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.7.0/client-libraries-cpp.md @@ -261,7 +261,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema -This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started). +This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started.md). ### Create producer with Avro schema diff --git a/site2/website/versioned_docs/version-2.7.0/client-libraries-dotnet.md b/site2/website/versioned_docs/version-2.7.0/client-libraries-dotnet.md index 4e0afe3d2ff6b..fbec5e473be69 100644 --- a/site2/website/versioned_docs/version-2.7.0/client-libraries-dotnet.md +++ b/site2/website/versioned_docs/version-2.7.0/client-libraries-dotnet.md @@ -170,7 +170,7 @@ var client = PulsarClient.Builder() Currently, the Pulsar C# client supports the TLS (Transport Layer Security) and JWT (JSON Web Token) authentication. -If you have followed [Authentication using TLS](security-tls-authentication), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: +If you have followed [Authentication using TLS](security-tls-authentication.md), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: 1. Create an unencrypted and password-less pfx file. diff --git a/site2/website/versioned_docs/version-2.7.0/client-libraries-go.md b/site2/website/versioned_docs/version-2.7.0/client-libraries-go.md index 97705a643c049..6c05e1d733422 100644 --- a/site2/website/versioned_docs/version-2.7.0/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.7.0/client-libraries-go.md @@ -35,7 +35,7 @@ import "github.com/apache/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -53,7 +53,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -680,7 +680,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker @@ -700,7 +700,7 @@ opts := pulsar.ClientOptions{ ## OAuth2 authentication -To use [OAuth2 authentication](security-oauth2), you'll need to configure your client to perform the following operations. +To use [OAuth2 authentication](security-oauth2.md), you'll need to configure your client to perform the following operations. This example shows how to configure OAuth2 authentication. ```go diff --git a/site2/website/versioned_docs/version-2.7.0/client-libraries-java.md b/site2/website/versioned_docs/version-2.7.0/client-libraries-java.md index bbe9473622fbe..ed7ce010e7550 100644 --- a/site2/website/versioned_docs/version-2.7.0/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.7.0/client-libraries-java.md @@ -5,7 +5,7 @@ sidebar_label: "Java" original_id: client-libraries-java --- -You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), and [reader](#reader) of a Java client are thread-safe. @@ -14,10 +14,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Includes both `pulsar-client` and `pulsar-client-admin`

    Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.

    In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -57,7 +57,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -83,7 +83,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -114,7 +114,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -237,7 +237,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -822,7 +822,7 @@ Total hash range size is 65536, so the max end of the range should be less than ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -936,7 +936,7 @@ The following schema formats are currently available for Java: ## Authentication -Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2). You can use the Pulsar Java client with all of them. +Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2.md). You can use the Pulsar Java client with all of them. ### TLS Authentication @@ -963,7 +963,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` @@ -1000,7 +1000,7 @@ PulsarClient client = PulsarClient.builder() ### Oauth2 -The following example shows how to use [Oauth2](security-oauth2) as an authentication provider for the Pulsar Java client. +The following example shows how to use [Oauth2](security-oauth2.md) as an authentication provider for the Pulsar Java client. You can use the factory method to configure authentication for Pulsar Java client. diff --git a/site2/website/versioned_docs/version-2.7.0/client-libraries-node.md b/site2/website/versioned_docs/version-2.7.0/client-libraries-node.md index 961114edb2de7..1ec0237974df6 100644 --- a/site2/website/versioned_docs/version-2.7.0/client-libraries-node.md +++ b/site2/website/versioned_docs/version-2.7.0/client-libraries-node.md @@ -46,7 +46,7 @@ Also, this library works only in Node.js 10.x or later because it uses the [`nod ::: ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here is an example for `localhost`: @@ -64,7 +64,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication), the URL will look like something like this: +If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication.md), the URL will look like something like this: ```http @@ -99,7 +99,7 @@ The following configurable parameters are available for Pulsar clients: | Parameter | Description | Default | | :-------- | :---------- | :------ | | `serviceUrl` | The connection URL for the Pulsar cluster. See [above](#connection-urls) for more info. | | -| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication) for more info. | | +| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication.md) for more info. | | | `operationTimeoutSeconds` | The timeout for Node.js client operations (creating producers, subscribing to and unsubscribing from [topics](reference-terminology.md#topic)). Retries will occur until this threshold is reached, at which point the operation will fail. | 30 | | `ioThreads` | The number of threads to use for handling connections to Pulsar [brokers](reference-terminology.md#broker). | 1 | | `messageListenerThreads` | The number of threads used by message listeners ([consumers](#consumers) and [readers](#readers)). | 1 | @@ -150,7 +150,7 @@ Pulsar Node.js producers have the following methods available: | :-------- | :---------- | :------ | | `topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer publishes messages. The topic format is `` or `//`. For example, `sample/ns1/my-topic`. | | | `producerName` | A name for the producer. If you do not explicitly assign a name, Pulsar will automatically generate a globally unique name. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | | -| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30000 | +| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30000 | | `initialSequenceId` | The initial sequence ID of the message. When producer send message, add sequence ID to message. The ID is increased each time to send. | | | `maxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `send` method will fail *unless* `blockIfQueueFull` is set to `true`. | 1000 | | `maxPendingMessagesAcrossPartitions` | The maximum size of the sum of partition's pending queue. | 50000 | diff --git a/site2/website/versioned_docs/version-2.7.0/client-libraries-python.md b/site2/website/versioned_docs/version-2.7.0/client-libraries-python.md index f9f9392ea68eb..5435a6a99a046 100644 --- a/site2/website/versioned_docs/version-2.7.0/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.7.0/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. diff --git a/site2/website/versioned_docs/version-2.7.0/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.7.0/client-libraries-websocket.md index cdf87d70aa905..4906a93ec0e6f 100644 --- a/site2/website/versioned_docs/version-2.7.0/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.7.0/client-libraries-websocket.md @@ -5,14 +5,14 @@ sidebar_label: "WebSocket" original_id: client-libraries-websocket --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.7.0/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.7.0/concepts-architecture-overview.md index c65f9149ceda5..ad3f04858b25a 100644 --- a/site2/website/versioned_docs/version-2.7.0/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.7.0/concepts-architecture-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Architecture" original_id: concepts-architecture-overview --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -17,20 +17,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -40,9 +40,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -130,17 +130,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -148,7 +148,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.7.0/concepts-authentication.md b/site2/website/versioned_docs/version-2.7.0/concepts-authentication.md index b375ecb16a2d8..f6307890c904a 100644 --- a/site2/website/versioned_docs/version-2.7.0/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.7.0/concepts-authentication.md @@ -5,5 +5,5 @@ sidebar_label: "Authentication and Authorization" original_id: concepts-authentication --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.7.0/concepts-clients.md b/site2/website/versioned_docs/version-2.7.0/concepts-clients.md index b68f76a2d8b08..4040624f7d636 100644 --- a/site2/website/versioned_docs/version-2.7.0/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.7.0/concepts-clients.md @@ -5,12 +5,12 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -38,7 +38,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.7.0/concepts-messaging.md b/site2/website/versioned_docs/version-2.7.0/concepts-messaging.md index a4ea0637c94eb..4b03824755273 100644 --- a/site2/website/versioned_docs/version-2.7.0/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.7.0/concepts-messaging.md @@ -21,8 +21,8 @@ Messages are the basic "unit" of Pulsar. The following table lists the component Component | Description :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer who produces the message. If you do not specify a producer name, the default name is used. Sequence ID | Each Pulsar message belongs to an ordered sequence on its topic. The sequence ID of the message is its order in that sequence. @@ -30,7 +30,7 @@ Publish time | The timestamp of when the message is published. The timestamp is Event time | An optional timestamp attached to a message by applications. For example, applications attach a timestamp on when the message is processed. If nothing is set to event time, the value is `0`. TypedMessageBuilder | It is used to construct a message. You can set message properties such as the message key, message value with `TypedMessageBuilder`.
    When you set `TypedMessageBuilder`, set the key as a string. If you set the key as other types, for example, an AVRO object, the key is sent as bytes, and it is difficult to get the AVRO object back on the consumer. -> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol). +> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -107,7 +107,7 @@ Messages are received from [brokers](reference-terminology.md#broker) either syn ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -344,7 +344,7 @@ A [subscription](#subscriptions) can have one or more consumers. When a consumer #### When to use -By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry). +By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry.md). #### How to use @@ -448,7 +448,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -460,7 +460,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -498,7 +498,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -553,7 +553,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -576,12 +576,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication.md). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.7.0/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.7.0/concepts-multi-tenancy.md index be752ccbbf9ca..c4acb21acb743 100644 --- a/site2/website/versioned_docs/version-2.7.0/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.7.0/concepts-multi-tenancy.md @@ -5,7 +5,7 @@ sidebar_label: "Multi Tenancy" original_id: concepts-multi-tenancy --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -21,7 +21,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -29,7 +29,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.7.0/concepts-overview.md b/site2/website/versioned_docs/version-2.7.0/concepts-overview.md index c1262b38daf16..e8a2f4b9d321a 100644 --- a/site2/website/versioned_docs/version-2.7.0/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.7.0/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.7.0/concepts-replication.md b/site2/website/versioned_docs/version-2.7.0/concepts-replication.md index 6e23962e99771..799f0eb4d92c6 100644 --- a/site2/website/versioned_docs/version-2.7.0/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.7.0/concepts-replication.md @@ -5,5 +5,5 @@ sidebar_label: "Geo Replication" original_id: concepts-replication --- -Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that. +Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo.md) in Pulsar enables you to do that. diff --git a/site2/website/versioned_docs/version-2.7.0/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.7.0/concepts-tiered-storage.md index 0b45b0a2ed501..f6988e53a8cd4 100644 --- a/site2/website/versioned_docs/version-2.7.0/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.7.0/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.7.0/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.7.0/concepts-topic-compaction.md index e402c965384fd..34b7ed7fbbd31 100644 --- a/site2/website/versioned_docs/version-2.7.0/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.7.0/concepts-topic-compaction.md @@ -7,7 +7,7 @@ original_id: concepts-topic-compaction Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -24,7 +24,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.7.0/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.7.0/cookbooks-compaction.md index 6fcbc197f9ecb..dfa314727241a 100644 --- a/site2/website/versioned_docs/version-2.7.0/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.7.0/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.7.0/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.7.0/cookbooks-non-persistent.md index 391569a8cbc6b..178301e86eb8d 100644 --- a/site2/website/versioned_docs/version-2.7.0/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.7.0/cookbooks-non-persistent.md @@ -41,7 +41,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -55,7 +55,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.7.0/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.7.0/cookbooks-partitioned.md index 7882fb9025312..fb9ac354cc6d6 100644 --- a/site2/website/versioned_docs/version-2.7.0/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.7.0/cookbooks-partitioned.md @@ -4,4 +4,4 @@ title: Partitioned topics sidebar_label: "Partitioned Topics" original_id: cookbooks-partitioned --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.7.0/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.7.0/cookbooks-retention-expiry.md index 5295b4c5de4ae..23175d607b37e 100644 --- a/site2/website/versioned_docs/version-2.7.0/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.7.0/cookbooks-retention-expiry.md @@ -22,7 +22,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.7.0/deploy-aws.md b/site2/website/versioned_docs/version-2.7.0/deploy-aws.md index 1c8f1a08d7249..662e99ceb9b09 100644 --- a/site2/website/versioned_docs/version-2.7.0/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.7.0/deploy-aws.md @@ -5,7 +5,7 @@ sidebar_label: "Amazon Web Services" original_id: deploy-aws --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -210,7 +210,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.7.0/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.7.0/deploy-bare-metal-multi-cluster.md index bdf8dfa251b3c..8d110c5d935bb 100644 --- a/site2/website/versioned_docs/version-2.7.0/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.7.0/deploy-bare-metal-multi-cluster.md @@ -9,17 +9,17 @@ original_id: deploy-bare-metal-multi-cluster 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using it in a startup or on a single team, you had better opt for a single cluster. For instructions on deploying a single cluster, -see the guide [here](deploy-bare-metal). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo). Deploying a multi-cluster Pulsar instance involves the following basic steps: +A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md). Deploying a multi-cluster Pulsar instance involves the following basic steps: * Deploying two separate [ZooKeeper](#deploy-zookeeper) quorums: a [local](#deploy-local-zookeeper) quorum for each cluster in the instance and a [configuration store](#configuration-store) quorum for instance-wide tasks * Initializing [cluster metadata](#cluster-metadata-initialization) for each cluster @@ -29,7 +29,7 @@ A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You If you want to deploy a single Pulsar cluster, see [Clusters and Brokers](getting-started-standalone.md#start-the-cluster). > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). ## System requirement Currently, Pulsar is available for 64-bit **macOS**, **Linux**, and **Windows**. To use Pulsar, you need to install 64-bit JRE/JDK 8 or later versions. @@ -67,7 +67,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -245,7 +245,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -364,17 +364,17 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). -You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > #### Service discovery already provided by many scheduling systems -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ### Service discovery setup -The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol) of Pulsar. +The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol.md) of Pulsar. To get started setting up the built-in service of discovery of Pulsar, you need to change a few parameters in the [`conf/discovery.conf`](reference-configuration.md#service-discovery) configuration file. Set the [`zookeeperServers`](reference-configuration.md#service-discovery-zookeeperServers) parameter to the ZooKeeper quorum connection string of the cluster and the [`configurationStoreServers`](reference-configuration.md#service-discovery-configurationStoreServers) setting to the [configuration store](reference-terminology.md#configuration-store) quorum connection string. @@ -399,7 +399,7 @@ $ bin/pulsar-daemon start discovery ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.7.0/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.7.0/deploy-bare-metal.md index 403f2fc6f79c7..9586bbfdedc30 100644 --- a/site2/website/versioned_docs/version-2.7.0/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.7.0/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using Pulsar in a startup or on a single team, it is simplest to opt for a single cluster. If you do need to run a multi-cluster Pulsar instance, -see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -101,7 +101,7 @@ The extracted directory contains the following subdirectories: Directory | Contains :---------|:-------- -`bin` |[command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` |[command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `data` | The data storage directory that ZooKeeper and BookKeeper use `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -263,9 +263,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (you had better not use a different port). -`--web-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). +`--web-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should not use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (you had better not use a different port). -`--broker-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). +`--broker-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). > If you do not have a DNS server, you can use multi-host format in the service URL with the following settings: @@ -405,7 +405,7 @@ webServicePortTls=8443 ### Enable Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -423,7 +423,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker). +If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker.md). ### Start Brokers diff --git a/site2/website/versioned_docs/version-2.7.0/deploy-dcos.md b/site2/website/versioned_docs/version-2.7.0/deploy-dcos.md index f5f8d1faa7dbe..952d5f47e30fa 100644 --- a/site2/website/versioned_docs/version-2.7.0/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.7.0/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -If you want to enable all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of +If you want to enable all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.7.0/deploy-docker.md b/site2/website/versioned_docs/version-2.7.0/deploy-docker.md index 64a0939aa787b..8348d78deb237 100644 --- a/site2/website/versioned_docs/version-2.7.0/deploy-docker.md +++ b/site2/website/versioned_docs/version-2.7.0/deploy-docker.md @@ -57,4 +57,4 @@ docker network connect pulsar broker To check whether the containers are successfully connected to the network, enter the `docker network inspect pulsar` command. -For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal). +For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal.md). diff --git a/site2/website/versioned_docs/version-2.7.0/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.7.0/deploy-kubernetes.md index dc7123d000467..1aefc6ad79f71 100644 --- a/site2/website/versioned_docs/version-2.7.0/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.7.0/deploy-kubernetes.md @@ -6,6 +6,6 @@ original_id: deploy-kubernetes --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.7.0/deploy-monitoring.md b/site2/website/versioned_docs/version-2.7.0/deploy-monitoring.md index 01915e05d83f7..2f4606ef6dcf5 100644 --- a/site2/website/versioned_docs/version-2.7.0/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.7.0/deploy-monitoring.md @@ -72,7 +72,7 @@ The default port for bookie is `8000`. You can change the port by configuring `p You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy). +When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy.md). ## Dashboards @@ -80,7 +80,7 @@ When you collect time series statistics, the major problem is to make sure the n ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager). +The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.7.0/develop-schema.md b/site2/website/versioned_docs/version-2.7.0/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.7.0/develop-schema.md +++ b/site2/website/versioned_docs/version-2.7.0/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.7.0/functions-deploy.md b/site2/website/versioned_docs/version-2.7.0/functions-deploy.md index 51f1140693dbe..d9496b3ed5b48 100644 --- a/site2/website/versioned_docs/version-2.7.0/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.7.0/functions-deploy.md @@ -9,12 +9,12 @@ original_id: functions-deploy To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface diff --git a/site2/website/versioned_docs/version-2.7.0/functions-develop.md b/site2/website/versioned_docs/version-2.7.0/functions-develop.md index 53c5dbd104153..2001909756ee7 100644 --- a/site2/website/versioned_docs/version-2.7.0/functions-develop.md +++ b/site2/website/versioned_docs/version-2.7.0/functions-develop.md @@ -896,7 +896,7 @@ public class MetricRecorderFunction implements Function { ``` -> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring) guide. +> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring.md) guide.     @@ -927,11 +927,11 @@ Currently, the feature is not available in Go. ```` ### Access metrics -To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring) in Pulsar. +To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring.md) in Pulsar. ## Security -If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker). For more details, refer to [Security settings](functions-worker.md#security-settings). +If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings). Pulsar Functions can support the following providers: diff --git a/site2/website/versioned_docs/version-2.7.0/functions-overview.md b/site2/website/versioned_docs/version-2.7.0/functions-overview.md index 25dc6029acf70..816d301e0fd0e 100644 --- a/site2/website/versioned_docs/version-2.7.0/functions-overview.md +++ b/site2/website/versioned_docs/version-2.7.0/functions-overview.md @@ -164,7 +164,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.7.0/functions-package.md b/site2/website/versioned_docs/version-2.7.0/functions-package.md index 82346aa8c5098..a995d5c158877 100644 --- a/site2/website/versioned_docs/version-2.7.0/functions-package.md +++ b/site2/website/versioned_docs/version-2.7.0/functions-package.md @@ -15,7 +15,7 @@ Currently, the window function is not available in Python and Go. ## Prerequisite -Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm). +Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm.md). To check whether the Docker image starts, you can use the `docker ps` command. diff --git a/site2/website/versioned_docs/version-2.7.0/functions-worker.md b/site2/website/versioned_docs/version-2.7.0/functions-worker.md index d06d8a53ae620..0addcb3648f4c 100644 --- a/site2/website/versioned_docs/version-2.7.0/functions-worker.md +++ b/site2/website/versioned_docs/version-2.7.0/functions-worker.md @@ -4,7 +4,7 @@ title: Deploy and manage functions worker sidebar_label: "Setup: Pulsar Functions Worker" original_id: functions-worker --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -179,7 +179,7 @@ brokerClientTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -199,7 +199,7 @@ authenticationProviders: [ provider1, provider2 ] ``` For *TLS Authentication* provider, follow the example below to add the necessary settings. -See [TLS Authentication](security-tls-authentication) for more details. +See [TLS Authentication](security-tls-authentication.md) for more details. ``` @@ -223,7 +223,7 @@ properties: ``` For *Token Authentication* provider, add necessary settings for `properties` if needed. -See [Token Authentication](security-jwt) for more details. +See [Token Authentication](security-jwt.md) for more details. ``` @@ -261,7 +261,7 @@ superUserRoles: You can use the public and private key pair that the application configures to perform encryption. Only the consumers with a valid key can decrypt the encrypted messages. -To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption). +To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption.md). We include the relevant configuration information of `CryptoConfig` into `ProducerConfig`. The specific configurable field information about `CryptoConfig` is as follows: diff --git a/site2/website/versioned_docs/version-2.7.0/getting-started-docker.md b/site2/website/versioned_docs/version-2.7.0/getting-started-docker.md index b9f12f0b52688..4f20971d75330 100644 --- a/site2/website/versioned_docs/version-2.7.0/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.7.0/getting-started-docker.md @@ -50,8 +50,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -110,7 +110,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.7.0/getting-started-helm.md b/site2/website/versioned_docs/version-2.7.0/getting-started-helm.md index 65a9fc04fe568..440087c275c05 100644 --- a/site2/website/versioned_docs/version-2.7.0/getting-started-helm.md +++ b/site2/website/versioned_docs/version-2.7.0/getting-started-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.7.0/getting-started-pulsar.md b/site2/website/versioned_docs/version-2.7.0/getting-started-pulsar.md index 11c5e66cb62c2..752590f57b558 100644 --- a/site2/website/versioned_docs/version-2.7.0/getting-started-pulsar.md +++ b/site2/website/versioned_docs/version-2.7.0/getting-started-pulsar.md @@ -5,13 +5,13 @@ sidebar_label: "Pulsar 2.0" original_id: pulsar-2.0 --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.7.0/getting-started-standalone.md b/site2/website/versioned_docs/version-2.7.0/getting-started-standalone.md index dad19d422f359..0921bb374b98b 100644 --- a/site2/website/versioned_docs/version-2.7.0/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.7.0/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -59,7 +59,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.7.0/helm-install.md b/site2/website/versioned_docs/version-2.7.0/helm-install.md index edb12260f010e..1f4d5eb69d5dd 100644 --- a/site2/website/versioned_docs/version-2.7.0/helm-install.md +++ b/site2/website/versioned_docs/version-2.7.0/helm-install.md @@ -21,7 +21,7 @@ Before deploying Pulsar, you need to prepare your environment. ### Tools -Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools) on your computer. +Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools.md) on your computer. ## Cloud cluster preparation @@ -37,8 +37,8 @@ To create and connect to the Kubernetes cluster, follow the instructions: ## Pulsar deployment -Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy). +Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy.md). ## Pulsar upgrade -To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade). +To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade.md). diff --git a/site2/website/versioned_docs/version-2.7.0/helm-overview.md b/site2/website/versioned_docs/version-2.7.0/helm-overview.md index 9cde85e2d18d5..385d535e319b6 100644 --- a/site2/website/versioned_docs/version-2.7.0/helm-overview.md +++ b/site2/website/versioned_docs/version-2.7.0/helm-overview.md @@ -52,9 +52,9 @@ It includes support for: ## Pulsar Helm chart quick start -To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install). +This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install.md). ## Troubleshooting @@ -64,7 +64,7 @@ We have done our best to make these charts as seamless as possible. Occasionally The Apache Pulsar Helm chart contains all required dependencies. -If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm) for your first iteration. +If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm.md) for your first iteration. 1. [Preparation](helm-prepare.md) 2. [Deployment](helm-deploy.md) @@ -82,7 +82,7 @@ helm upgrade apache/pulsar -f pulsar.yaml ``` -For more detailed information, see [Upgrading](helm-upgrade). +For more detailed information, see [Upgrading](helm-upgrade.md). ## Uninstallation diff --git a/site2/website/versioned_docs/version-2.7.0/helm-prepare.md b/site2/website/versioned_docs/version-2.7.0/helm-prepare.md index 95d7e668f9eca..5e9f2f9ef4f68 100644 --- a/site2/website/versioned_docs/version-2.7.0/helm-prepare.md +++ b/site2/website/versioned_docs/version-2.7.0/helm-prepare.md @@ -89,4 +89,4 @@ PROJECT= USE_LOCAL_SSD=true LOCAL_SSD_COUNT= ## Next Steps -Continue with the [installation of the chart](helm-deploy) once you have the cluster up and running. +Continue with the [installation of the chart](helm-deploy.md) once you have the cluster up and running. diff --git a/site2/website/versioned_docs/version-2.7.0/helm-tools.md b/site2/website/versioned_docs/version-2.7.0/helm-tools.md index efd61eae32506..6ba89006913b6 100644 --- a/site2/website/versioned_docs/version-2.7.0/helm-tools.md +++ b/site2/website/versioned_docs/version-2.7.0/helm-tools.md @@ -25,7 +25,7 @@ You can get Helm from the project's [releases page](https://github.com/helm/helm ### Next steps -Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare). +Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare.md). ## Additional information diff --git a/site2/website/versioned_docs/version-2.7.0/helm-upgrade.md b/site2/website/versioned_docs/version-2.7.0/helm-upgrade.md index e39cf97407355..7d671e6bfb3c1 100644 --- a/site2/website/versioned_docs/version-2.7.0/helm-upgrade.md +++ b/site2/website/versioned_docs/version-2.7.0/helm-upgrade.md @@ -20,7 +20,7 @@ You can retrieve your previous `--set` arguments cleanly, with `helm get values To upgrade Apache Pulsar to a newer version, follow these steps: 1. Check the change log for the specific version you would like to upgrade to. -2. Go through [deployment documentation](helm-deploy) step by step. +2. Go through [deployment documentation](helm-deploy.md) step by step. 3. Extract your previous `--set` arguments with the following command. ```bash diff --git a/site2/website/versioned_docs/version-2.7.0/io-cdc.md b/site2/website/versioned_docs/version-2.7.0/io-cdc.md index 20f16ae0d6904..e6e662884826d 100644 --- a/site2/website/versioned_docs/version-2.7.0/io-cdc.md +++ b/site2/website/versioned_docs/version-2.7.0/io-cdc.md @@ -13,8 +13,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.7.0/io-develop.md b/site2/website/versioned_docs/version-2.7.0/io-develop.md index 4a815b3b9d6f6..7a356662560c9 100644 --- a/site2/website/versioned_docs/version-2.7.0/io-develop.md +++ b/site2/website/versioned_docs/version-2.7.0/io-develop.md @@ -8,15 +8,15 @@ original_id: io-develop This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -163,7 +163,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -192,7 +192,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). diff --git a/site2/website/versioned_docs/version-2.7.0/io-overview.md b/site2/website/versioned_docs/version-2.7.0/io-overview.md index 68df2472d6334..b538604edc46f 100644 --- a/site2/website/versioned_docs/version-2.7.0/io-overview.md +++ b/site2/website/versioned_docs/version-2.7.0/io-overview.md @@ -158,7 +158,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.7.0/io-quickstart.md b/site2/website/versioned_docs/version-2.7.0/io-quickstart.md index b8fea1560bdeb..8474c93f51336 100644 --- a/site2/website/versioned_docs/version-2.7.0/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.7.0/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you are able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you are able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However, all the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -109,7 +109,7 @@ This section demonstrates how to connect Pulsar to Cassandra. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink.md). ::: @@ -236,11 +236,11 @@ You can create a configuration file through one of the following methods. ``` -For more information, see [Cassandra sink connector](io-cassandra-sink). +For more information, see [Cassandra sink connector](io-cassandra-sink.md). ### Create a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on them. Run the following command to create a Cassandra sink connector with sink type _cassandra_ and the config file _examples/cassandra-sink.yml_ created previously. @@ -267,7 +267,7 @@ as a Pulsar Function and writes the messages produced in the topic _test_cassand ### Inspect a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * Get the information of a Cassandra sink. @@ -431,7 +431,7 @@ to monitor a connector and perform other operations on it. ### Delete a Cassandra Sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. ```bash @@ -454,7 +454,7 @@ This section demonstrates how to connect Pulsar to PostgreSQL. ::: ->For more information, see [JDBC sink connector](io-jdbc-sink). +>For more information, see [JDBC sink connector](io-jdbc-sink.md). ### Setup a PostgreSQL cluster @@ -614,7 +614,7 @@ In this section, you need to configure a JDBC sink connector. ### Create a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on it. This example creates a sink connector and specifies the desired information. @@ -660,7 +660,7 @@ The sink has been created successfully if the following message appears. ### Inspect a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * List all running JDBC sink(s). @@ -780,7 +780,7 @@ to monitor a connector and perform other operations on it. ### Stop a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to stop a connector and perform other operations on it. ```bash @@ -808,7 +808,7 @@ The sink instance has been stopped successfully if the following message disappe ### Restart a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to restart a connector and perform other operations on it. ```bash @@ -844,7 +844,7 @@ Note that `pulsar-admin sinks localrun options` **runs a sink connector locally* ### Update a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to update a connector and perform other operations on it. This example updates the parallelism of the _pulsar-postgres-jdbc-sink_ sink connector to 2. @@ -912,7 +912,7 @@ The result shows that the parallelism is 2. ### Delete a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. This example deletes the _pulsar-postgres-jdbc-sink_ sink connector. diff --git a/site2/website/versioned_docs/version-2.7.0/io-use.md b/site2/website/versioned_docs/version-2.7.0/io-use.md index b6409d4420ad7..da9ed746c4d37 100644 --- a/site2/website/versioned_docs/version-2.7.0/io-use.md +++ b/site2/website/versioned_docs/version-2.7.0/io-use.md @@ -15,7 +15,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.7.0/kubernetes-helm.md b/site2/website/versioned_docs/version-2.7.0/kubernetes-helm.md index a9f768e6a96b3..ea92a0968cd7d 100644 --- a/site2/website/versioned_docs/version-2.7.0/kubernetes-helm.md +++ b/site2/website/versioned_docs/version-2.7.0/kubernetes-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.7.0/reference-cli-tools.md b/site2/website/versioned_docs/version-2.7.0/reference-cli-tools.md index 74c628427c3b1..73e092f0b9d9a 100644 --- a/site2/website/versioned_docs/version-2.7.0/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.7.0/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.7.0/schema-get-started.md b/site2/website/versioned_docs/version-2.7.0/schema-get-started.md index 97e270586a26d..afacb0fa51f2e 100644 --- a/site2/website/versioned_docs/version-2.7.0/schema-get-started.md +++ b/site2/website/versioned_docs/version-2.7.0/schema-get-started.md @@ -17,7 +17,7 @@ Applications typically adopt one of the following approaches to guarantee type s #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ### Client-side approach diff --git a/site2/website/versioned_docs/version-2.7.0/schema-manage.md b/site2/website/versioned_docs/version-2.7.0/schema-manage.md index 662b8b523726d..84d0ccdc23164 100644 --- a/site2/website/versioned_docs/version-2.7.0/schema-manage.md +++ b/site2/website/versioned_docs/version-2.7.0/schema-manage.md @@ -160,7 +160,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.7.0/schema-understand.md b/site2/website/versioned_docs/version-2.7.0/schema-understand.md index 333c902b58319..d17c6168da59f 100644 --- a/site2/website/versioned_docs/version-2.7.0/schema-understand.md +++ b/site2/website/versioned_docs/version-2.7.0/schema-understand.md @@ -380,7 +380,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```java diff --git a/site2/website/versioned_docs/version-2.7.0/security-athenz.md b/site2/website/versioned_docs/version-2.7.0/security-athenz.md index 947c3f470be46..8a39fe25316d0 100644 --- a/site2/website/versioned_docs/version-2.7.0/security-athenz.md +++ b/site2/website/versioned_docs/version-2.7.0/security-athenz.md @@ -76,7 +76,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.7.0/security-authorization.md b/site2/website/versioned_docs/version-2.7.0/security-authorization.md index e6785873f0e10..7ac09b6f439ea 100644 --- a/site2/website/versioned_docs/version-2.7.0/security-authorization.md +++ b/site2/website/versioned_docs/version-2.7.0/security-authorization.md @@ -27,7 +27,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -59,7 +59,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -87,7 +87,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication diff --git a/site2/website/versioned_docs/version-2.7.0/security-bouncy-castle.md b/site2/website/versioned_docs/version-2.7.0/security-bouncy-castle.md index abc25c71b8ae2..be937055d8e31 100644 --- a/site2/website/versioned_docs/version-2.7.0/security-bouncy-castle.md +++ b/site2/website/versioned_docs/version-2.7.0/security-bouncy-castle.md @@ -16,7 +16,7 @@ In Pulsar, security and crypto have dependencies on BouncyCastle Jars. For the d `Bouncy Castle` provides both [FIPS](https://www.bouncycastle.org/fips_faq.html) and non-FIPS version. But in a JVM, you can not include both of the 2 versions, and you need to exclude the current version before include the other. -In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. +In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption.md). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. ## How BouncyCastle modules packaged in Pulsar diff --git a/site2/website/versioned_docs/version-2.7.0/security-extending.md b/site2/website/versioned_docs/version-2.7.0/security-extending.md index c088e3a2c1c2c..e7484453b8beb 100644 --- a/site2/website/versioned_docs/version-2.7.0/security-extending.md +++ b/site2/website/versioned_docs/version-2.7.0/security-extending.md @@ -9,7 +9,7 @@ Pulsar provides a way to use custom authentication and authorization mechanisms. ## Authentication -Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview). +Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview.md). You can use a custom authentication mechanism by providing the implementation in the form of two plugins. One plugin is for the Client library and the other plugin is for the Pulsar Proxy and/or Pulsar Broker to validate the credentials. diff --git a/site2/website/versioned_docs/version-2.7.0/security-jwt.md b/site2/website/versioned_docs/version-2.7.0/security-jwt.md index 546e72449165a..17ea2cbf6228c 100644 --- a/site2/website/versioned_docs/version-2.7.0/security-jwt.md +++ b/site2/website/versioned_docs/version-2.7.0/security-jwt.md @@ -32,11 +32,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -303,7 +303,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.7.0/security-kerberos.md b/site2/website/versioned_docs/version-2.7.0/security-kerberos.md index 670586a9d1691..c49fa3bea1fce 100644 --- a/site2/website/versioned_docs/version-2.7.0/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.7.0/security-kerberos.md @@ -373,7 +373,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.7.0/security-overview.md b/site2/website/versioned_docs/version-2.7.0/security-overview.md index c7823104d145b..a1664fcb8b450 100644 --- a/site2/website/versioned_docs/version-2.7.0/security-overview.md +++ b/site2/website/versioned_docs/version-2.7.0/security-overview.md @@ -21,7 +21,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.7.0/security-tls-authentication.md b/site2/website/versioned_docs/version-2.7.0/security-tls-authentication.md index 20a6604a6c1fe..e1b67ffa05b78 100644 --- a/site2/website/versioned_docs/version-2.7.0/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.7.0/security-tls-authentication.md @@ -7,9 +7,9 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -101,7 +101,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -121,7 +121,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.7.0/security-tls-keystore.md b/site2/website/versioned_docs/version-2.7.0/security-tls-keystore.md index e7c18c025cc15..d5204593cfe58 100644 --- a/site2/website/versioned_docs/version-2.7.0/security-tls-keystore.md +++ b/site2/website/versioned_docs/version-2.7.0/security-tls-keystore.md @@ -7,7 +7,7 @@ original_id: security-tls-keystore ## Overview -Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication) between clients and Apache Pulsar service. +Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication.md) between clients and Apache Pulsar service. By default it uses PEM format file configuration. This page tries to describe use [KeyStore](https://en.wikipedia.org/wiki/Java_KeyStore) type configure for TLS. @@ -165,7 +165,7 @@ This is similar to [TLS encryption configuing for client with PEM type](security For a a minimal configuration, user need to provide the TrustStore information. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties @@ -258,7 +258,7 @@ webSocketServiceEnabled=false Besides the TLS encryption configuring. The main work is configuring the KeyStore, which contains a valid CN as client role, for client. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties diff --git a/site2/website/versioned_docs/version-2.7.0/security-tls-transport.md b/site2/website/versioned_docs/version-2.7.0/security-tls-transport.md index 083aca48f65e2..52709e4867032 100644 --- a/site2/website/versioned_docs/version-2.7.0/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.7.0/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,9 +27,9 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). -`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ## Create TLS certificates @@ -130,7 +130,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Broker Configuration -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -201,7 +201,7 @@ The examples below show that hostname verification is disabled for the CLI tools ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.7.0/security-token-admin.md b/site2/website/versioned_docs/version-2.7.0/security-token-admin.md index 6bef23b106415..39bb6178e2bf3 100644 --- a/site2/website/versioned_docs/version-2.7.0/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.7.0/security-token-admin.md @@ -163,7 +163,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.7.0/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.7.0/sql-deployment-configurations.md index c6d7c88a07e94..cc8bdbcd4fd51 100644 --- a/site2/website/versioned_docs/version-2.7.0/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.7.0/sql-deployment-configurations.md @@ -64,7 +64,7 @@ Since Pulsar SQL is powered by [Presto](https://prestosql.io), the configuration :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.7.0/sql-overview.md b/site2/website/versioned_docs/version-2.7.0/sql-overview.md index 2bb78260265b2..8abb23ea2acbb 100644 --- a/site2/website/versioned_docs/version-2.7.0/sql-overview.md +++ b/site2/website/versioned_docs/version-2.7.0/sql-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Overview" original_id: sql-overview --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.7.0/standalone-docker.md b/site2/website/versioned_docs/version-2.7.0/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.7.0/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.7.0/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.7.0/standalone.md b/site2/website/versioned_docs/version-2.7.0/standalone.md index 220af52ce6861..e91f94140a4ac 100644 --- a/site2/website/versioned_docs/version-2.7.0/standalone.md +++ b/site2/website/versioned_docs/version-2.7.0/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -59,7 +59,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.7.0/tiered-storage-overview.md b/site2/website/versioned_docs/version-2.7.0/tiered-storage-overview.md index 7c6528ce2b36c..c635034f463b4 100644 --- a/site2/website/versioned_docs/version-2.7.0/tiered-storage-overview.md +++ b/site2/website/versioned_docs/version-2.7.0/tiered-storage-overview.md @@ -13,9 +13,9 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws). + - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws.md). - - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs). + - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs.md). ::: @@ -25,7 +25,7 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem). + For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem.md). ::: diff --git a/site2/website/versioned_docs/version-2.7.1/admin-api-brokers.md b/site2/website/versioned_docs/version-2.7.1/admin-api-brokers.md index 751bc14983205..61047c4d631d1 100644 --- a/site2/website/versioned_docs/version-2.7.1/admin-api-brokers.md +++ b/site2/website/versioned_docs/version-2.7.1/admin-api-brokers.md @@ -18,7 +18,7 @@ Pulsar brokers consist of two components: [Brokers](reference-terminology.md#broker) can be managed via: -* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/brokers` endpoint of the admin {@inject: rest:REST:/} API * The `brokers` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin.html} object in the [Java API](client-libraries-java.md) @@ -123,7 +123,7 @@ One way to configure a Pulsar [broker](reference-terminology.md#broker) is to su But since all broker configuration in Pulsar is stored in ZooKeeper, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values. -* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. +* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. * In the Pulsar admin {@inject: rest:REST:/} API, dynamic configuration is managed through the `/admin/v2/brokers/configuration` endpoint. ### Update dynamic configuration diff --git a/site2/website/versioned_docs/version-2.7.1/admin-api-clusters.md b/site2/website/versioned_docs/version-2.7.1/admin-api-clusters.md index 3eb238506cc69..e0e9fb5f91f65 100644 --- a/site2/website/versioned_docs/version-2.7.1/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.7.1/admin-api-clusters.md @@ -16,7 +16,7 @@ servers (aka [bookies](reference-terminology.md#bookie)), and a [ZooKeeper](http Clusters can be managed via: -* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/clusters` endpoint of the admin {@inject: rest:REST:/} API * The `clusters` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md) @@ -79,7 +79,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -103,11 +103,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block -If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: +If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin.md) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: |Name|Description|Default| |----|-----------|-------| diff --git a/site2/website/versioned_docs/version-2.7.1/admin-api-partitioned-topics.md b/site2/website/versioned_docs/version-2.7.1/admin-api-partitioned-topics.md index 6734586da1718..5ce182282e032 100644 --- a/site2/website/versioned_docs/version-2.7.1/admin-api-partitioned-topics.md +++ b/site2/website/versioned_docs/version-2.7.1/admin-api-partitioned-topics.md @@ -5,4 +5,4 @@ sidebar_label: "Partitioned topics" original_id: admin-api-partitioned-topics --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.7.1/admin-api-persistent-topics.md b/site2/website/versioned_docs/version-2.7.1/admin-api-persistent-topics.md index b6d293b1e6be5..50d135b72f542 100644 --- a/site2/website/versioned_docs/version-2.7.1/admin-api-persistent-topics.md +++ b/site2/website/versioned_docs/version-2.7.1/admin-api-persistent-topics.md @@ -5,4 +5,4 @@ sidebar_label: "Persistent topics" original_id: admin-api-persistent-topics --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.7.1/admin-api-tenants.md b/site2/website/versioned_docs/version-2.7.1/admin-api-tenants.md index 63afe58620c62..d78aa2e55f4c3 100644 --- a/site2/website/versioned_docs/version-2.7.1/admin-api-tenants.md +++ b/site2/website/versioned_docs/version-2.7.1/admin-api-tenants.md @@ -11,7 +11,7 @@ import TabItem from '@theme/TabItem'; ```` -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -108,7 +108,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.7.1/client-libraries-cgo.md b/site2/website/versioned_docs/version-2.7.1/client-libraries-cgo.md index c79f7bb965473..f352f942b7714 100644 --- a/site2/website/versioned_docs/version-2.7.1/client-libraries-cgo.md +++ b/site2/website/versioned_docs/version-2.7.1/client-libraries-cgo.md @@ -24,7 +24,7 @@ Currently, the following Go clients are maintained in two repositories. ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Install go package @@ -57,7 +57,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -75,7 +75,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -236,14 +236,14 @@ Parameter | Description | Default `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | `Properties`| Attach a set of application defined properties to the producer. This properties will be visible in the topic stats | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | Set the number of max pending messages across all the partitions. This setting will be used to lower the max pending messages for each partition `MaxPendingMessages(int)`, if the total exceeds the configured value.| `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | `Batching` | Control whether automatic batching of messages is enabled for the producer. | false `BatchingMaxPublishDelay` | Set the time period within which the messages sent will be batched (default: 1ms) if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or until | 1ms `BatchingMaxMessages` | Set the maximum number of messages permitted in a batch. (default: 1000) If set to a value greater than 1, messages will be queued until this threshold is reached or batch interval has elapsed | 1000 @@ -512,7 +512,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.7.1/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.7.1/client-libraries-cpp.md index bb52d9effa404..684df5fd2d825 100644 --- a/site2/website/versioned_docs/version-2.7.1/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.7.1/client-libraries-cpp.md @@ -261,7 +261,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema -This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started). +This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started.md). ### Create producer with Avro schema diff --git a/site2/website/versioned_docs/version-2.7.1/client-libraries-dotnet.md b/site2/website/versioned_docs/version-2.7.1/client-libraries-dotnet.md index ade664cd95277..b574fa0b2e5ed 100644 --- a/site2/website/versioned_docs/version-2.7.1/client-libraries-dotnet.md +++ b/site2/website/versioned_docs/version-2.7.1/client-libraries-dotnet.md @@ -170,7 +170,7 @@ var client = PulsarClient.Builder() Currently, the Pulsar C# client supports the TLS (Transport Layer Security) and JWT (JSON Web Token) authentication. -If you have followed [Authentication using TLS](security-tls-authentication), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: +If you have followed [Authentication using TLS](security-tls-authentication.md), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: 1. Create an unencrypted and password-less pfx file. diff --git a/site2/website/versioned_docs/version-2.7.1/client-libraries-go.md b/site2/website/versioned_docs/version-2.7.1/client-libraries-go.md index 97705a643c049..6c05e1d733422 100644 --- a/site2/website/versioned_docs/version-2.7.1/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.7.1/client-libraries-go.md @@ -35,7 +35,7 @@ import "github.com/apache/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -53,7 +53,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -680,7 +680,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker @@ -700,7 +700,7 @@ opts := pulsar.ClientOptions{ ## OAuth2 authentication -To use [OAuth2 authentication](security-oauth2), you'll need to configure your client to perform the following operations. +To use [OAuth2 authentication](security-oauth2.md), you'll need to configure your client to perform the following operations. This example shows how to configure OAuth2 authentication. ```go diff --git a/site2/website/versioned_docs/version-2.7.1/client-libraries-java.md b/site2/website/versioned_docs/version-2.7.1/client-libraries-java.md index bbe9473622fbe..3fdcbe21cfda0 100644 --- a/site2/website/versioned_docs/version-2.7.1/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.7.1/client-libraries-java.md @@ -5,7 +5,7 @@ sidebar_label: "Java" original_id: client-libraries-java --- -You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), and [reader](#reader) of a Java client are thread-safe. @@ -14,10 +14,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Includes both `pulsar-client` and `pulsar-client-admin`

    Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.

    In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -57,7 +57,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -83,7 +83,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -114,7 +114,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -237,7 +237,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -822,7 +822,7 @@ Total hash range size is 65536, so the max end of the range should be less than ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -936,7 +936,7 @@ The following schema formats are currently available for Java: ## Authentication -Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2). You can use the Pulsar Java client with all of them. +Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2.md). You can use the Pulsar Java client with all of them. ### TLS Authentication @@ -963,7 +963,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` @@ -1000,7 +1000,7 @@ PulsarClient client = PulsarClient.builder() ### Oauth2 -The following example shows how to use [Oauth2](security-oauth2) as an authentication provider for the Pulsar Java client. +The following example shows how to use [Oauth2](security-oauth2.md) as an authentication provider for the Pulsar Java client. You can use the factory method to configure authentication for Pulsar Java client. diff --git a/site2/website/versioned_docs/version-2.7.1/client-libraries-node.md b/site2/website/versioned_docs/version-2.7.1/client-libraries-node.md index 961114edb2de7..1ec0237974df6 100644 --- a/site2/website/versioned_docs/version-2.7.1/client-libraries-node.md +++ b/site2/website/versioned_docs/version-2.7.1/client-libraries-node.md @@ -46,7 +46,7 @@ Also, this library works only in Node.js 10.x or later because it uses the [`nod ::: ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here is an example for `localhost`: @@ -64,7 +64,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication), the URL will look like something like this: +If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication.md), the URL will look like something like this: ```http @@ -99,7 +99,7 @@ The following configurable parameters are available for Pulsar clients: | Parameter | Description | Default | | :-------- | :---------- | :------ | | `serviceUrl` | The connection URL for the Pulsar cluster. See [above](#connection-urls) for more info. | | -| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication) for more info. | | +| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication.md) for more info. | | | `operationTimeoutSeconds` | The timeout for Node.js client operations (creating producers, subscribing to and unsubscribing from [topics](reference-terminology.md#topic)). Retries will occur until this threshold is reached, at which point the operation will fail. | 30 | | `ioThreads` | The number of threads to use for handling connections to Pulsar [brokers](reference-terminology.md#broker). | 1 | | `messageListenerThreads` | The number of threads used by message listeners ([consumers](#consumers) and [readers](#readers)). | 1 | @@ -150,7 +150,7 @@ Pulsar Node.js producers have the following methods available: | :-------- | :---------- | :------ | | `topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer publishes messages. The topic format is `` or `//`. For example, `sample/ns1/my-topic`. | | | `producerName` | A name for the producer. If you do not explicitly assign a name, Pulsar will automatically generate a globally unique name. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | | -| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30000 | +| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30000 | | `initialSequenceId` | The initial sequence ID of the message. When producer send message, add sequence ID to message. The ID is increased each time to send. | | | `maxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `send` method will fail *unless* `blockIfQueueFull` is set to `true`. | 1000 | | `maxPendingMessagesAcrossPartitions` | The maximum size of the sum of partition's pending queue. | 50000 | diff --git a/site2/website/versioned_docs/version-2.7.1/client-libraries-python.md b/site2/website/versioned_docs/version-2.7.1/client-libraries-python.md index 1d2d8e4767341..b2c401049705f 100644 --- a/site2/website/versioned_docs/version-2.7.1/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.7.1/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. diff --git a/site2/website/versioned_docs/version-2.7.1/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.7.1/client-libraries-websocket.md index cdf87d70aa905..4906a93ec0e6f 100644 --- a/site2/website/versioned_docs/version-2.7.1/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.7.1/client-libraries-websocket.md @@ -5,14 +5,14 @@ sidebar_label: "WebSocket" original_id: client-libraries-websocket --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.7.1/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.7.1/concepts-architecture-overview.md index c65f9149ceda5..ad3f04858b25a 100644 --- a/site2/website/versioned_docs/version-2.7.1/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.7.1/concepts-architecture-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Architecture" original_id: concepts-architecture-overview --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -17,20 +17,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -40,9 +40,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -130,17 +130,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -148,7 +148,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.7.1/concepts-authentication.md b/site2/website/versioned_docs/version-2.7.1/concepts-authentication.md index b375ecb16a2d8..f6307890c904a 100644 --- a/site2/website/versioned_docs/version-2.7.1/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.7.1/concepts-authentication.md @@ -5,5 +5,5 @@ sidebar_label: "Authentication and Authorization" original_id: concepts-authentication --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.7.1/concepts-clients.md b/site2/website/versioned_docs/version-2.7.1/concepts-clients.md index b68f76a2d8b08..4040624f7d636 100644 --- a/site2/website/versioned_docs/version-2.7.1/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.7.1/concepts-clients.md @@ -5,12 +5,12 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -38,7 +38,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.7.1/concepts-messaging.md b/site2/website/versioned_docs/version-2.7.1/concepts-messaging.md index a4ea0637c94eb..4b03824755273 100644 --- a/site2/website/versioned_docs/version-2.7.1/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.7.1/concepts-messaging.md @@ -21,8 +21,8 @@ Messages are the basic "unit" of Pulsar. The following table lists the component Component | Description :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer who produces the message. If you do not specify a producer name, the default name is used. Sequence ID | Each Pulsar message belongs to an ordered sequence on its topic. The sequence ID of the message is its order in that sequence. @@ -30,7 +30,7 @@ Publish time | The timestamp of when the message is published. The timestamp is Event time | An optional timestamp attached to a message by applications. For example, applications attach a timestamp on when the message is processed. If nothing is set to event time, the value is `0`. TypedMessageBuilder | It is used to construct a message. You can set message properties such as the message key, message value with `TypedMessageBuilder`.
    When you set `TypedMessageBuilder`, set the key as a string. If you set the key as other types, for example, an AVRO object, the key is sent as bytes, and it is difficult to get the AVRO object back on the consumer. -> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol). +> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -107,7 +107,7 @@ Messages are received from [brokers](reference-terminology.md#broker) either syn ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -344,7 +344,7 @@ A [subscription](#subscriptions) can have one or more consumers. When a consumer #### When to use -By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry). +By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry.md). #### How to use @@ -448,7 +448,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -460,7 +460,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -498,7 +498,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -553,7 +553,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -576,12 +576,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication.md). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.7.1/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.7.1/concepts-multi-tenancy.md index be752ccbbf9ca..c4acb21acb743 100644 --- a/site2/website/versioned_docs/version-2.7.1/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.7.1/concepts-multi-tenancy.md @@ -5,7 +5,7 @@ sidebar_label: "Multi Tenancy" original_id: concepts-multi-tenancy --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -21,7 +21,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -29,7 +29,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.7.1/concepts-overview.md b/site2/website/versioned_docs/version-2.7.1/concepts-overview.md index c1262b38daf16..e8a2f4b9d321a 100644 --- a/site2/website/versioned_docs/version-2.7.1/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.7.1/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.7.1/concepts-replication.md b/site2/website/versioned_docs/version-2.7.1/concepts-replication.md index 6e23962e99771..799f0eb4d92c6 100644 --- a/site2/website/versioned_docs/version-2.7.1/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.7.1/concepts-replication.md @@ -5,5 +5,5 @@ sidebar_label: "Geo Replication" original_id: concepts-replication --- -Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that. +Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo.md) in Pulsar enables you to do that. diff --git a/site2/website/versioned_docs/version-2.7.1/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.7.1/concepts-tiered-storage.md index 0b45b0a2ed501..f6988e53a8cd4 100644 --- a/site2/website/versioned_docs/version-2.7.1/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.7.1/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.7.1/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.7.1/concepts-topic-compaction.md index e402c965384fd..34b7ed7fbbd31 100644 --- a/site2/website/versioned_docs/version-2.7.1/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.7.1/concepts-topic-compaction.md @@ -7,7 +7,7 @@ original_id: concepts-topic-compaction Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -24,7 +24,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.7.1/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.7.1/cookbooks-compaction.md index 6fcbc197f9ecb..dfa314727241a 100644 --- a/site2/website/versioned_docs/version-2.7.1/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.7.1/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.7.1/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.7.1/cookbooks-non-persistent.md index 391569a8cbc6b..178301e86eb8d 100644 --- a/site2/website/versioned_docs/version-2.7.1/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.7.1/cookbooks-non-persistent.md @@ -41,7 +41,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -55,7 +55,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.7.1/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.7.1/cookbooks-partitioned.md index 7882fb9025312..fb9ac354cc6d6 100644 --- a/site2/website/versioned_docs/version-2.7.1/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.7.1/cookbooks-partitioned.md @@ -4,4 +4,4 @@ title: Partitioned topics sidebar_label: "Partitioned Topics" original_id: cookbooks-partitioned --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.7.1/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.7.1/cookbooks-retention-expiry.md index 5295b4c5de4ae..23175d607b37e 100644 --- a/site2/website/versioned_docs/version-2.7.1/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.7.1/cookbooks-retention-expiry.md @@ -22,7 +22,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.7.1/deploy-aws.md b/site2/website/versioned_docs/version-2.7.1/deploy-aws.md index 071effeffde24..93c389b56e2cf 100644 --- a/site2/website/versioned_docs/version-2.7.1/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.7.1/deploy-aws.md @@ -5,7 +5,7 @@ sidebar_label: "Amazon Web Services" original_id: deploy-aws --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -210,7 +210,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors) , edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md) , edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.7.1/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.7.1/deploy-bare-metal-multi-cluster.md index bdf8dfa251b3c..8d110c5d935bb 100644 --- a/site2/website/versioned_docs/version-2.7.1/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.7.1/deploy-bare-metal-multi-cluster.md @@ -9,17 +9,17 @@ original_id: deploy-bare-metal-multi-cluster 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using it in a startup or on a single team, you had better opt for a single cluster. For instructions on deploying a single cluster, -see the guide [here](deploy-bare-metal). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo). Deploying a multi-cluster Pulsar instance involves the following basic steps: +A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md). Deploying a multi-cluster Pulsar instance involves the following basic steps: * Deploying two separate [ZooKeeper](#deploy-zookeeper) quorums: a [local](#deploy-local-zookeeper) quorum for each cluster in the instance and a [configuration store](#configuration-store) quorum for instance-wide tasks * Initializing [cluster metadata](#cluster-metadata-initialization) for each cluster @@ -29,7 +29,7 @@ A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You If you want to deploy a single Pulsar cluster, see [Clusters and Brokers](getting-started-standalone.md#start-the-cluster). > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). ## System requirement Currently, Pulsar is available for 64-bit **macOS**, **Linux**, and **Windows**. To use Pulsar, you need to install 64-bit JRE/JDK 8 or later versions. @@ -67,7 +67,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -245,7 +245,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -364,17 +364,17 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). -You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > #### Service discovery already provided by many scheduling systems -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ### Service discovery setup -The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol) of Pulsar. +The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol.md) of Pulsar. To get started setting up the built-in service of discovery of Pulsar, you need to change a few parameters in the [`conf/discovery.conf`](reference-configuration.md#service-discovery) configuration file. Set the [`zookeeperServers`](reference-configuration.md#service-discovery-zookeeperServers) parameter to the ZooKeeper quorum connection string of the cluster and the [`configurationStoreServers`](reference-configuration.md#service-discovery-configurationStoreServers) setting to the [configuration store](reference-terminology.md#configuration-store) quorum connection string. @@ -399,7 +399,7 @@ $ bin/pulsar-daemon start discovery ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.7.1/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.7.1/deploy-bare-metal.md index 403f2fc6f79c7..9586bbfdedc30 100644 --- a/site2/website/versioned_docs/version-2.7.1/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.7.1/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using Pulsar in a startup or on a single team, it is simplest to opt for a single cluster. If you do need to run a multi-cluster Pulsar instance, -see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -101,7 +101,7 @@ The extracted directory contains the following subdirectories: Directory | Contains :---------|:-------- -`bin` |[command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` |[command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `data` | The data storage directory that ZooKeeper and BookKeeper use `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -263,9 +263,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (you had better not use a different port). -`--web-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). +`--web-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should not use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (you had better not use a different port). -`--broker-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). +`--broker-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). > If you do not have a DNS server, you can use multi-host format in the service URL with the following settings: @@ -405,7 +405,7 @@ webServicePortTls=8443 ### Enable Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -423,7 +423,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker). +If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker.md). ### Start Brokers diff --git a/site2/website/versioned_docs/version-2.7.1/deploy-dcos.md b/site2/website/versioned_docs/version-2.7.1/deploy-dcos.md index f5f8d1faa7dbe..952d5f47e30fa 100644 --- a/site2/website/versioned_docs/version-2.7.1/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.7.1/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -If you want to enable all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of +If you want to enable all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.7.1/deploy-docker.md b/site2/website/versioned_docs/version-2.7.1/deploy-docker.md index 64a0939aa787b..8348d78deb237 100644 --- a/site2/website/versioned_docs/version-2.7.1/deploy-docker.md +++ b/site2/website/versioned_docs/version-2.7.1/deploy-docker.md @@ -57,4 +57,4 @@ docker network connect pulsar broker To check whether the containers are successfully connected to the network, enter the `docker network inspect pulsar` command. -For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal). +For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal.md). diff --git a/site2/website/versioned_docs/version-2.7.1/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.7.1/deploy-kubernetes.md index dc7123d000467..1aefc6ad79f71 100644 --- a/site2/website/versioned_docs/version-2.7.1/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.7.1/deploy-kubernetes.md @@ -6,6 +6,6 @@ original_id: deploy-kubernetes --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.7.1/deploy-monitoring.md b/site2/website/versioned_docs/version-2.7.1/deploy-monitoring.md index ab3735a0829b5..efc4e3c8ad9df 100644 --- a/site2/website/versioned_docs/version-2.7.1/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.7.1/deploy-monitoring.md @@ -86,7 +86,7 @@ Those metrics are added in the Prometheus interface, you can monitor and check t You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy). +When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy.md). ## Dashboards @@ -94,7 +94,7 @@ When you collect time series statistics, the major problem is to make sure the n ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager). +The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.7.1/develop-schema.md b/site2/website/versioned_docs/version-2.7.1/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.7.1/develop-schema.md +++ b/site2/website/versioned_docs/version-2.7.1/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.7.1/functions-deploy.md b/site2/website/versioned_docs/version-2.7.1/functions-deploy.md index 51f1140693dbe..d9496b3ed5b48 100644 --- a/site2/website/versioned_docs/version-2.7.1/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.7.1/functions-deploy.md @@ -9,12 +9,12 @@ original_id: functions-deploy To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface diff --git a/site2/website/versioned_docs/version-2.7.1/functions-develop.md b/site2/website/versioned_docs/version-2.7.1/functions-develop.md index 512a62529ed74..f916eb81341c9 100644 --- a/site2/website/versioned_docs/version-2.7.1/functions-develop.md +++ b/site2/website/versioned_docs/version-2.7.1/functions-develop.md @@ -958,7 +958,7 @@ public class MetricRecorderFunction implements Function { ``` -> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring) guide. +> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring.md) guide.     @@ -989,11 +989,11 @@ Currently, the feature is not available in Go. ```` ### Access metrics -To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring) in Pulsar. +To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring.md) in Pulsar. ## Security -If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker). For more details, refer to [Security settings](functions-worker.md#security-settings). +If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings). Pulsar Functions can support the following providers: diff --git a/site2/website/versioned_docs/version-2.7.1/functions-overview.md b/site2/website/versioned_docs/version-2.7.1/functions-overview.md index 25dc6029acf70..816d301e0fd0e 100644 --- a/site2/website/versioned_docs/version-2.7.1/functions-overview.md +++ b/site2/website/versioned_docs/version-2.7.1/functions-overview.md @@ -164,7 +164,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.7.1/functions-package.md b/site2/website/versioned_docs/version-2.7.1/functions-package.md index 82346aa8c5098..a995d5c158877 100644 --- a/site2/website/versioned_docs/version-2.7.1/functions-package.md +++ b/site2/website/versioned_docs/version-2.7.1/functions-package.md @@ -15,7 +15,7 @@ Currently, the window function is not available in Python and Go. ## Prerequisite -Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm). +Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm.md). To check whether the Docker image starts, you can use the `docker ps` command. diff --git a/site2/website/versioned_docs/version-2.7.1/functions-worker.md b/site2/website/versioned_docs/version-2.7.1/functions-worker.md index fa1467aa5fd30..9b603e88de609 100644 --- a/site2/website/versioned_docs/version-2.7.1/functions-worker.md +++ b/site2/website/versioned_docs/version-2.7.1/functions-worker.md @@ -4,7 +4,7 @@ title: Deploy and manage functions worker sidebar_label: "Setup: Pulsar Functions Worker" original_id: functions-worker --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -180,7 +180,7 @@ brokerClientTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -200,7 +200,7 @@ authenticationProviders: [ provider1, provider2 ] ``` For *TLS Authentication* provider, follow the example below to add the necessary settings. -See [TLS Authentication](security-tls-authentication) for more details. +See [TLS Authentication](security-tls-authentication.md) for more details. ``` @@ -224,7 +224,7 @@ properties: ``` For *Token Authentication* provider, add necessary settings for `properties` if needed. -See [Token Authentication](security-jwt) for more details. +See [Token Authentication](security-jwt.md) for more details. ``` @@ -262,7 +262,7 @@ superUserRoles: You can use the public and private key pair that the application configures to perform encryption. Only the consumers with a valid key can decrypt the encrypted messages. -To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption). +To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption.md). We include the relevant configuration information of `CryptoConfig` into `ProducerConfig`. The specific configurable field information about `CryptoConfig` is as follows: diff --git a/site2/website/versioned_docs/version-2.7.1/getting-started-docker.md b/site2/website/versioned_docs/version-2.7.1/getting-started-docker.md index b9f12f0b52688..4f20971d75330 100644 --- a/site2/website/versioned_docs/version-2.7.1/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.7.1/getting-started-docker.md @@ -50,8 +50,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -110,7 +110,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.7.1/getting-started-helm.md b/site2/website/versioned_docs/version-2.7.1/getting-started-helm.md index 65a9fc04fe568..440087c275c05 100644 --- a/site2/website/versioned_docs/version-2.7.1/getting-started-helm.md +++ b/site2/website/versioned_docs/version-2.7.1/getting-started-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.7.1/getting-started-pulsar.md b/site2/website/versioned_docs/version-2.7.1/getting-started-pulsar.md index 11c5e66cb62c2..752590f57b558 100644 --- a/site2/website/versioned_docs/version-2.7.1/getting-started-pulsar.md +++ b/site2/website/versioned_docs/version-2.7.1/getting-started-pulsar.md @@ -5,13 +5,13 @@ sidebar_label: "Pulsar 2.0" original_id: pulsar-2.0 --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.7.1/getting-started-standalone.md b/site2/website/versioned_docs/version-2.7.1/getting-started-standalone.md index dad19d422f359..0921bb374b98b 100644 --- a/site2/website/versioned_docs/version-2.7.1/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.7.1/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -59,7 +59,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.7.1/helm-install.md b/site2/website/versioned_docs/version-2.7.1/helm-install.md index edb12260f010e..1f4d5eb69d5dd 100644 --- a/site2/website/versioned_docs/version-2.7.1/helm-install.md +++ b/site2/website/versioned_docs/version-2.7.1/helm-install.md @@ -21,7 +21,7 @@ Before deploying Pulsar, you need to prepare your environment. ### Tools -Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools) on your computer. +Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools.md) on your computer. ## Cloud cluster preparation @@ -37,8 +37,8 @@ To create and connect to the Kubernetes cluster, follow the instructions: ## Pulsar deployment -Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy). +Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy.md). ## Pulsar upgrade -To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade). +To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade.md). diff --git a/site2/website/versioned_docs/version-2.7.1/helm-overview.md b/site2/website/versioned_docs/version-2.7.1/helm-overview.md index 9cde85e2d18d5..385d535e319b6 100644 --- a/site2/website/versioned_docs/version-2.7.1/helm-overview.md +++ b/site2/website/versioned_docs/version-2.7.1/helm-overview.md @@ -52,9 +52,9 @@ It includes support for: ## Pulsar Helm chart quick start -To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install). +This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install.md). ## Troubleshooting @@ -64,7 +64,7 @@ We have done our best to make these charts as seamless as possible. Occasionally The Apache Pulsar Helm chart contains all required dependencies. -If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm) for your first iteration. +If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm.md) for your first iteration. 1. [Preparation](helm-prepare.md) 2. [Deployment](helm-deploy.md) @@ -82,7 +82,7 @@ helm upgrade apache/pulsar -f pulsar.yaml ``` -For more detailed information, see [Upgrading](helm-upgrade). +For more detailed information, see [Upgrading](helm-upgrade.md). ## Uninstallation diff --git a/site2/website/versioned_docs/version-2.7.1/helm-prepare.md b/site2/website/versioned_docs/version-2.7.1/helm-prepare.md index 95d7e668f9eca..5e9f2f9ef4f68 100644 --- a/site2/website/versioned_docs/version-2.7.1/helm-prepare.md +++ b/site2/website/versioned_docs/version-2.7.1/helm-prepare.md @@ -89,4 +89,4 @@ PROJECT= USE_LOCAL_SSD=true LOCAL_SSD_COUNT= ## Next Steps -Continue with the [installation of the chart](helm-deploy) once you have the cluster up and running. +Continue with the [installation of the chart](helm-deploy.md) once you have the cluster up and running. diff --git a/site2/website/versioned_docs/version-2.7.1/helm-tools.md b/site2/website/versioned_docs/version-2.7.1/helm-tools.md index efd61eae32506..6ba89006913b6 100644 --- a/site2/website/versioned_docs/version-2.7.1/helm-tools.md +++ b/site2/website/versioned_docs/version-2.7.1/helm-tools.md @@ -25,7 +25,7 @@ You can get Helm from the project's [releases page](https://github.com/helm/helm ### Next steps -Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare). +Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare.md). ## Additional information diff --git a/site2/website/versioned_docs/version-2.7.1/helm-upgrade.md b/site2/website/versioned_docs/version-2.7.1/helm-upgrade.md index e39cf97407355..7d671e6bfb3c1 100644 --- a/site2/website/versioned_docs/version-2.7.1/helm-upgrade.md +++ b/site2/website/versioned_docs/version-2.7.1/helm-upgrade.md @@ -20,7 +20,7 @@ You can retrieve your previous `--set` arguments cleanly, with `helm get values To upgrade Apache Pulsar to a newer version, follow these steps: 1. Check the change log for the specific version you would like to upgrade to. -2. Go through [deployment documentation](helm-deploy) step by step. +2. Go through [deployment documentation](helm-deploy.md) step by step. 3. Extract your previous `--set` arguments with the following command. ```bash diff --git a/site2/website/versioned_docs/version-2.7.1/io-cdc.md b/site2/website/versioned_docs/version-2.7.1/io-cdc.md index 20f16ae0d6904..e6e662884826d 100644 --- a/site2/website/versioned_docs/version-2.7.1/io-cdc.md +++ b/site2/website/versioned_docs/version-2.7.1/io-cdc.md @@ -13,8 +13,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.7.1/io-develop.md b/site2/website/versioned_docs/version-2.7.1/io-develop.md index 4a815b3b9d6f6..7a356662560c9 100644 --- a/site2/website/versioned_docs/version-2.7.1/io-develop.md +++ b/site2/website/versioned_docs/version-2.7.1/io-develop.md @@ -8,15 +8,15 @@ original_id: io-develop This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -163,7 +163,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -192,7 +192,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). diff --git a/site2/website/versioned_docs/version-2.7.1/io-overview.md b/site2/website/versioned_docs/version-2.7.1/io-overview.md index 68df2472d6334..b538604edc46f 100644 --- a/site2/website/versioned_docs/version-2.7.1/io-overview.md +++ b/site2/website/versioned_docs/version-2.7.1/io-overview.md @@ -158,7 +158,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.7.1/io-quickstart.md b/site2/website/versioned_docs/version-2.7.1/io-quickstart.md index b8fea1560bdeb..8474c93f51336 100644 --- a/site2/website/versioned_docs/version-2.7.1/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.7.1/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you are able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you are able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However, all the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -109,7 +109,7 @@ This section demonstrates how to connect Pulsar to Cassandra. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink.md). ::: @@ -236,11 +236,11 @@ You can create a configuration file through one of the following methods. ``` -For more information, see [Cassandra sink connector](io-cassandra-sink). +For more information, see [Cassandra sink connector](io-cassandra-sink.md). ### Create a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on them. Run the following command to create a Cassandra sink connector with sink type _cassandra_ and the config file _examples/cassandra-sink.yml_ created previously. @@ -267,7 +267,7 @@ as a Pulsar Function and writes the messages produced in the topic _test_cassand ### Inspect a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * Get the information of a Cassandra sink. @@ -431,7 +431,7 @@ to monitor a connector and perform other operations on it. ### Delete a Cassandra Sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. ```bash @@ -454,7 +454,7 @@ This section demonstrates how to connect Pulsar to PostgreSQL. ::: ->For more information, see [JDBC sink connector](io-jdbc-sink). +>For more information, see [JDBC sink connector](io-jdbc-sink.md). ### Setup a PostgreSQL cluster @@ -614,7 +614,7 @@ In this section, you need to configure a JDBC sink connector. ### Create a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on it. This example creates a sink connector and specifies the desired information. @@ -660,7 +660,7 @@ The sink has been created successfully if the following message appears. ### Inspect a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * List all running JDBC sink(s). @@ -780,7 +780,7 @@ to monitor a connector and perform other operations on it. ### Stop a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to stop a connector and perform other operations on it. ```bash @@ -808,7 +808,7 @@ The sink instance has been stopped successfully if the following message disappe ### Restart a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to restart a connector and perform other operations on it. ```bash @@ -844,7 +844,7 @@ Note that `pulsar-admin sinks localrun options` **runs a sink connector locally* ### Update a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to update a connector and perform other operations on it. This example updates the parallelism of the _pulsar-postgres-jdbc-sink_ sink connector to 2. @@ -912,7 +912,7 @@ The result shows that the parallelism is 2. ### Delete a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. This example deletes the _pulsar-postgres-jdbc-sink_ sink connector. diff --git a/site2/website/versioned_docs/version-2.7.1/io-use.md b/site2/website/versioned_docs/version-2.7.1/io-use.md index b6409d4420ad7..da9ed746c4d37 100644 --- a/site2/website/versioned_docs/version-2.7.1/io-use.md +++ b/site2/website/versioned_docs/version-2.7.1/io-use.md @@ -15,7 +15,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.7.1/kubernetes-helm.md b/site2/website/versioned_docs/version-2.7.1/kubernetes-helm.md index a9f768e6a96b3..ea92a0968cd7d 100644 --- a/site2/website/versioned_docs/version-2.7.1/kubernetes-helm.md +++ b/site2/website/versioned_docs/version-2.7.1/kubernetes-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.7.1/reference-cli-tools.md b/site2/website/versioned_docs/version-2.7.1/reference-cli-tools.md index d17bd3efd9790..2bc3cc56130f5 100644 --- a/site2/website/versioned_docs/version-2.7.1/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.7.1/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.7.1/schema-get-started.md b/site2/website/versioned_docs/version-2.7.1/schema-get-started.md index 97e270586a26d..afacb0fa51f2e 100644 --- a/site2/website/versioned_docs/version-2.7.1/schema-get-started.md +++ b/site2/website/versioned_docs/version-2.7.1/schema-get-started.md @@ -17,7 +17,7 @@ Applications typically adopt one of the following approaches to guarantee type s #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ### Client-side approach diff --git a/site2/website/versioned_docs/version-2.7.1/schema-manage.md b/site2/website/versioned_docs/version-2.7.1/schema-manage.md index 4ae363882b2d5..c588aae619eee 100644 --- a/site2/website/versioned_docs/version-2.7.1/schema-manage.md +++ b/site2/website/versioned_docs/version-2.7.1/schema-manage.md @@ -160,7 +160,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.7.1/schema-understand.md b/site2/website/versioned_docs/version-2.7.1/schema-understand.md index 333c902b58319..d17c6168da59f 100644 --- a/site2/website/versioned_docs/version-2.7.1/schema-understand.md +++ b/site2/website/versioned_docs/version-2.7.1/schema-understand.md @@ -380,7 +380,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```java diff --git a/site2/website/versioned_docs/version-2.7.1/security-athenz.md b/site2/website/versioned_docs/version-2.7.1/security-athenz.md index 947c3f470be46..8a39fe25316d0 100644 --- a/site2/website/versioned_docs/version-2.7.1/security-athenz.md +++ b/site2/website/versioned_docs/version-2.7.1/security-athenz.md @@ -76,7 +76,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.7.1/security-authorization.md b/site2/website/versioned_docs/version-2.7.1/security-authorization.md index e6785873f0e10..7ac09b6f439ea 100644 --- a/site2/website/versioned_docs/version-2.7.1/security-authorization.md +++ b/site2/website/versioned_docs/version-2.7.1/security-authorization.md @@ -27,7 +27,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -59,7 +59,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -87,7 +87,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication diff --git a/site2/website/versioned_docs/version-2.7.1/security-bouncy-castle.md b/site2/website/versioned_docs/version-2.7.1/security-bouncy-castle.md index abc25c71b8ae2..be937055d8e31 100644 --- a/site2/website/versioned_docs/version-2.7.1/security-bouncy-castle.md +++ b/site2/website/versioned_docs/version-2.7.1/security-bouncy-castle.md @@ -16,7 +16,7 @@ In Pulsar, security and crypto have dependencies on BouncyCastle Jars. For the d `Bouncy Castle` provides both [FIPS](https://www.bouncycastle.org/fips_faq.html) and non-FIPS version. But in a JVM, you can not include both of the 2 versions, and you need to exclude the current version before include the other. -In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. +In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption.md). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. ## How BouncyCastle modules packaged in Pulsar diff --git a/site2/website/versioned_docs/version-2.7.1/security-extending.md b/site2/website/versioned_docs/version-2.7.1/security-extending.md index c088e3a2c1c2c..e7484453b8beb 100644 --- a/site2/website/versioned_docs/version-2.7.1/security-extending.md +++ b/site2/website/versioned_docs/version-2.7.1/security-extending.md @@ -9,7 +9,7 @@ Pulsar provides a way to use custom authentication and authorization mechanisms. ## Authentication -Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview). +Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview.md). You can use a custom authentication mechanism by providing the implementation in the form of two plugins. One plugin is for the Client library and the other plugin is for the Pulsar Proxy and/or Pulsar Broker to validate the credentials. diff --git a/site2/website/versioned_docs/version-2.7.1/security-jwt.md b/site2/website/versioned_docs/version-2.7.1/security-jwt.md index 656ab362f71b0..91117bedff753 100644 --- a/site2/website/versioned_docs/version-2.7.1/security-jwt.md +++ b/site2/website/versioned_docs/version-2.7.1/security-jwt.md @@ -32,11 +32,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -303,7 +303,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.7.1/security-kerberos.md b/site2/website/versioned_docs/version-2.7.1/security-kerberos.md index 670586a9d1691..c49fa3bea1fce 100644 --- a/site2/website/versioned_docs/version-2.7.1/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.7.1/security-kerberos.md @@ -373,7 +373,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.7.1/security-overview.md b/site2/website/versioned_docs/version-2.7.1/security-overview.md index c7823104d145b..a1664fcb8b450 100644 --- a/site2/website/versioned_docs/version-2.7.1/security-overview.md +++ b/site2/website/versioned_docs/version-2.7.1/security-overview.md @@ -21,7 +21,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.7.1/security-tls-authentication.md b/site2/website/versioned_docs/version-2.7.1/security-tls-authentication.md index 79aa8ea9fd13d..61c8aa43e1421 100644 --- a/site2/website/versioned_docs/version-2.7.1/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.7.1/security-tls-authentication.md @@ -7,9 +7,9 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -101,7 +101,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -121,7 +121,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.7.1/security-tls-keystore.md b/site2/website/versioned_docs/version-2.7.1/security-tls-keystore.md index 062d903e0e2ca..cb14afd1e4080 100644 --- a/site2/website/versioned_docs/version-2.7.1/security-tls-keystore.md +++ b/site2/website/versioned_docs/version-2.7.1/security-tls-keystore.md @@ -7,7 +7,7 @@ original_id: security-tls-keystore ## Overview -Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication) between clients and Apache Pulsar service. +Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication.md) between clients and Apache Pulsar service. By default it uses PEM format file configuration. This page tries to describe use [KeyStore](https://en.wikipedia.org/wiki/Java_KeyStore) type configure for TLS. @@ -165,7 +165,7 @@ This is similar to [TLS encryption configuing for client with PEM type](security For a a minimal configuration, user need to provide the TrustStore information. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties @@ -258,7 +258,7 @@ webSocketServiceEnabled=false Besides the TLS encryption configuring. The main work is configuring the KeyStore, which contains a valid CN as client role, for client. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties diff --git a/site2/website/versioned_docs/version-2.7.1/security-tls-transport.md b/site2/website/versioned_docs/version-2.7.1/security-tls-transport.md index 083aca48f65e2..52709e4867032 100644 --- a/site2/website/versioned_docs/version-2.7.1/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.7.1/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,9 +27,9 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). -`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ## Create TLS certificates @@ -130,7 +130,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Broker Configuration -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -201,7 +201,7 @@ The examples below show that hostname verification is disabled for the CLI tools ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.7.1/security-token-admin.md b/site2/website/versioned_docs/version-2.7.1/security-token-admin.md index 2173a6cb1d11d..a5d69da9f0602 100644 --- a/site2/website/versioned_docs/version-2.7.1/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.7.1/security-token-admin.md @@ -163,7 +163,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.7.1/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.7.1/sql-deployment-configurations.md index c6d7c88a07e94..cc8bdbcd4fd51 100644 --- a/site2/website/versioned_docs/version-2.7.1/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.7.1/sql-deployment-configurations.md @@ -64,7 +64,7 @@ Since Pulsar SQL is powered by [Presto](https://prestosql.io), the configuration :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.7.1/sql-overview.md b/site2/website/versioned_docs/version-2.7.1/sql-overview.md index 2bb78260265b2..8abb23ea2acbb 100644 --- a/site2/website/versioned_docs/version-2.7.1/sql-overview.md +++ b/site2/website/versioned_docs/version-2.7.1/sql-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Overview" original_id: sql-overview --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.7.1/standalone-docker.md b/site2/website/versioned_docs/version-2.7.1/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.7.1/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.7.1/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.7.1/standalone.md b/site2/website/versioned_docs/version-2.7.1/standalone.md index 220af52ce6861..e91f94140a4ac 100644 --- a/site2/website/versioned_docs/version-2.7.1/standalone.md +++ b/site2/website/versioned_docs/version-2.7.1/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -59,7 +59,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.7.1/tiered-storage-overview.md b/site2/website/versioned_docs/version-2.7.1/tiered-storage-overview.md index 846a5a5387bcb..f9e8341d68194 100644 --- a/site2/website/versioned_docs/version-2.7.1/tiered-storage-overview.md +++ b/site2/website/versioned_docs/version-2.7.1/tiered-storage-overview.md @@ -13,8 +13,8 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws). - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs). + For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws.md). + For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs.md). ::: @@ -24,7 +24,7 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem). + For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem.md). ::: diff --git a/site2/website/versioned_docs/version-2.7.2/admin-api-brokers.md b/site2/website/versioned_docs/version-2.7.2/admin-api-brokers.md index 751bc14983205..61047c4d631d1 100644 --- a/site2/website/versioned_docs/version-2.7.2/admin-api-brokers.md +++ b/site2/website/versioned_docs/version-2.7.2/admin-api-brokers.md @@ -18,7 +18,7 @@ Pulsar brokers consist of two components: [Brokers](reference-terminology.md#broker) can be managed via: -* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/brokers` endpoint of the admin {@inject: rest:REST:/} API * The `brokers` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin.html} object in the [Java API](client-libraries-java.md) @@ -123,7 +123,7 @@ One way to configure a Pulsar [broker](reference-terminology.md#broker) is to su But since all broker configuration in Pulsar is stored in ZooKeeper, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values. -* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. +* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. * In the Pulsar admin {@inject: rest:REST:/} API, dynamic configuration is managed through the `/admin/v2/brokers/configuration` endpoint. ### Update dynamic configuration diff --git a/site2/website/versioned_docs/version-2.7.2/admin-api-clusters.md b/site2/website/versioned_docs/version-2.7.2/admin-api-clusters.md index f3af175295514..79508763e1db0 100644 --- a/site2/website/versioned_docs/version-2.7.2/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.7.2/admin-api-clusters.md @@ -16,7 +16,7 @@ servers (aka [bookies](reference-terminology.md#bookie)), and a [ZooKeeper](http Clusters can be managed via: -* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/clusters` endpoint of the admin {@inject: rest:REST:/} API * The `clusters` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md) @@ -79,7 +79,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -103,11 +103,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block -If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: +If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin.md) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: |Name|Description|Default| |----|-----------|-------| diff --git a/site2/website/versioned_docs/version-2.7.2/admin-api-partitioned-topics.md b/site2/website/versioned_docs/version-2.7.2/admin-api-partitioned-topics.md index 28338e8cc4a69..4582ba14772d8 100644 --- a/site2/website/versioned_docs/version-2.7.2/admin-api-partitioned-topics.md +++ b/site2/website/versioned_docs/version-2.7.2/admin-api-partitioned-topics.md @@ -5,4 +5,4 @@ sidebar_label: "Partitioned topics" original_id: admin-api-partitioned-topics --- -For details of the content, refer to [manage topics](admin-api-topics). +For details of the content, refer to [manage topics](admin-api-topics.md). diff --git a/site2/website/versioned_docs/version-2.7.2/admin-api-persistent-topics.md b/site2/website/versioned_docs/version-2.7.2/admin-api-persistent-topics.md index 596e0358227ea..13e24f1d9eba4 100644 --- a/site2/website/versioned_docs/version-2.7.2/admin-api-persistent-topics.md +++ b/site2/website/versioned_docs/version-2.7.2/admin-api-persistent-topics.md @@ -5,4 +5,4 @@ sidebar_label: "Persistent topics" original_id: admin-api-persistent-topics --- -For details of the content, refer to [manage topics](admin-api-topics). +For details of the content, refer to [manage topics](admin-api-topics.md). diff --git a/site2/website/versioned_docs/version-2.7.2/admin-api-tenants.md b/site2/website/versioned_docs/version-2.7.2/admin-api-tenants.md index 63afe58620c62..d78aa2e55f4c3 100644 --- a/site2/website/versioned_docs/version-2.7.2/admin-api-tenants.md +++ b/site2/website/versioned_docs/version-2.7.2/admin-api-tenants.md @@ -11,7 +11,7 @@ import TabItem from '@theme/TabItem'; ```` -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -108,7 +108,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.7.2/client-libraries-cgo.md b/site2/website/versioned_docs/version-2.7.2/client-libraries-cgo.md index c79f7bb965473..f352f942b7714 100644 --- a/site2/website/versioned_docs/version-2.7.2/client-libraries-cgo.md +++ b/site2/website/versioned_docs/version-2.7.2/client-libraries-cgo.md @@ -24,7 +24,7 @@ Currently, the following Go clients are maintained in two repositories. ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Install go package @@ -57,7 +57,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -75,7 +75,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -236,14 +236,14 @@ Parameter | Description | Default `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | `Properties`| Attach a set of application defined properties to the producer. This properties will be visible in the topic stats | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | Set the number of max pending messages across all the partitions. This setting will be used to lower the max pending messages for each partition `MaxPendingMessages(int)`, if the total exceeds the configured value.| `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | `Batching` | Control whether automatic batching of messages is enabled for the producer. | false `BatchingMaxPublishDelay` | Set the time period within which the messages sent will be batched (default: 1ms) if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or until | 1ms `BatchingMaxMessages` | Set the maximum number of messages permitted in a batch. (default: 1000) If set to a value greater than 1, messages will be queued until this threshold is reached or batch interval has elapsed | 1000 @@ -512,7 +512,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.7.2/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.7.2/client-libraries-cpp.md index bb52d9effa404..684df5fd2d825 100644 --- a/site2/website/versioned_docs/version-2.7.2/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.7.2/client-libraries-cpp.md @@ -261,7 +261,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema -This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started). +This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started.md). ### Create producer with Avro schema diff --git a/site2/website/versioned_docs/version-2.7.2/client-libraries-dotnet.md b/site2/website/versioned_docs/version-2.7.2/client-libraries-dotnet.md index 4e0afe3d2ff6b..fbec5e473be69 100644 --- a/site2/website/versioned_docs/version-2.7.2/client-libraries-dotnet.md +++ b/site2/website/versioned_docs/version-2.7.2/client-libraries-dotnet.md @@ -170,7 +170,7 @@ var client = PulsarClient.Builder() Currently, the Pulsar C# client supports the TLS (Transport Layer Security) and JWT (JSON Web Token) authentication. -If you have followed [Authentication using TLS](security-tls-authentication), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: +If you have followed [Authentication using TLS](security-tls-authentication.md), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: 1. Create an unencrypted and password-less pfx file. diff --git a/site2/website/versioned_docs/version-2.7.2/client-libraries-go.md b/site2/website/versioned_docs/version-2.7.2/client-libraries-go.md index 97705a643c049..6c05e1d733422 100644 --- a/site2/website/versioned_docs/version-2.7.2/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.7.2/client-libraries-go.md @@ -35,7 +35,7 @@ import "github.com/apache/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -53,7 +53,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -680,7 +680,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker @@ -700,7 +700,7 @@ opts := pulsar.ClientOptions{ ## OAuth2 authentication -To use [OAuth2 authentication](security-oauth2), you'll need to configure your client to perform the following operations. +To use [OAuth2 authentication](security-oauth2.md), you'll need to configure your client to perform the following operations. This example shows how to configure OAuth2 authentication. ```go diff --git a/site2/website/versioned_docs/version-2.7.2/client-libraries-java.md b/site2/website/versioned_docs/version-2.7.2/client-libraries-java.md index bbe9473622fbe..ed7ce010e7550 100644 --- a/site2/website/versioned_docs/version-2.7.2/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.7.2/client-libraries-java.md @@ -5,7 +5,7 @@ sidebar_label: "Java" original_id: client-libraries-java --- -You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), and [reader](#reader) of a Java client are thread-safe. @@ -14,10 +14,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Includes both `pulsar-client` and `pulsar-client-admin`

    Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.

    In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -57,7 +57,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -83,7 +83,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -114,7 +114,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -237,7 +237,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -822,7 +822,7 @@ Total hash range size is 65536, so the max end of the range should be less than ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -936,7 +936,7 @@ The following schema formats are currently available for Java: ## Authentication -Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2). You can use the Pulsar Java client with all of them. +Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2.md). You can use the Pulsar Java client with all of them. ### TLS Authentication @@ -963,7 +963,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` @@ -1000,7 +1000,7 @@ PulsarClient client = PulsarClient.builder() ### Oauth2 -The following example shows how to use [Oauth2](security-oauth2) as an authentication provider for the Pulsar Java client. +The following example shows how to use [Oauth2](security-oauth2.md) as an authentication provider for the Pulsar Java client. You can use the factory method to configure authentication for Pulsar Java client. diff --git a/site2/website/versioned_docs/version-2.7.2/client-libraries-node.md b/site2/website/versioned_docs/version-2.7.2/client-libraries-node.md index 89fd4850f014d..1af24dfd1e456 100644 --- a/site2/website/versioned_docs/version-2.7.2/client-libraries-node.md +++ b/site2/website/versioned_docs/version-2.7.2/client-libraries-node.md @@ -46,7 +46,7 @@ Also, this library works only in Node.js 10.x or later because it uses the [`nod ::: ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here is an example for `localhost`: @@ -64,7 +64,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication), the URL will look like something like this: +If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication.md), the URL will look like something like this: ```http @@ -99,7 +99,7 @@ The following configurable parameters are available for Pulsar clients: | Parameter | Description | Default | | :-------- | :---------- | :------ | | `serviceUrl` | The connection URL for the Pulsar cluster. See [above](#connection-urls) for more info. | | -| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication) for more info. | | +| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication.md) for more info. | | | `operationTimeoutSeconds` | The timeout for Node.js client operations (creating producers, subscribing to and unsubscribing from [topics](reference-terminology.md#topic)). Retries will occur until this threshold is reached, at which point the operation will fail. | 30 | | `ioThreads` | The number of threads to use for handling connections to Pulsar [brokers](reference-terminology.md#broker). | 1 | | `messageListenerThreads` | The number of threads used by message listeners ([consumers](#consumers) and [readers](#readers)). | 1 | @@ -150,7 +150,7 @@ Pulsar Node.js producers have the following methods available: | :-------- | :---------- | :------ | | `topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer publishes messages. The topic format is `` or `//`. For example, `sample/ns1/my-topic`. | | | `producerName` | A name for the producer. If you do not explicitly assign a name, Pulsar will automatically generate a globally unique name. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | | -| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30000 | +| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30000 | | `initialSequenceId` | The initial sequence ID of the message. When producer send message, add sequence ID to message. The ID is increased each time to send. | | | `maxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `send` method will fail *unless* `blockIfQueueFull` is set to `true`. | 1000 | | `maxPendingMessagesAcrossPartitions` | The maximum size of the sum of partition's pending queue. | 50000 | diff --git a/site2/website/versioned_docs/version-2.7.2/client-libraries-python.md b/site2/website/versioned_docs/version-2.7.2/client-libraries-python.md index 7d44db909ca9f..e7ea7f01813f8 100644 --- a/site2/website/versioned_docs/version-2.7.2/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.7.2/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. diff --git a/site2/website/versioned_docs/version-2.7.2/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.7.2/client-libraries-websocket.md index cdf87d70aa905..4906a93ec0e6f 100644 --- a/site2/website/versioned_docs/version-2.7.2/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.7.2/client-libraries-websocket.md @@ -5,14 +5,14 @@ sidebar_label: "WebSocket" original_id: client-libraries-websocket --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.7.2/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.7.2/concepts-architecture-overview.md index c65f9149ceda5..ad3f04858b25a 100644 --- a/site2/website/versioned_docs/version-2.7.2/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.7.2/concepts-architecture-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Architecture" original_id: concepts-architecture-overview --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -17,20 +17,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -40,9 +40,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -130,17 +130,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -148,7 +148,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.7.2/concepts-authentication.md b/site2/website/versioned_docs/version-2.7.2/concepts-authentication.md index b375ecb16a2d8..f6307890c904a 100644 --- a/site2/website/versioned_docs/version-2.7.2/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.7.2/concepts-authentication.md @@ -5,5 +5,5 @@ sidebar_label: "Authentication and Authorization" original_id: concepts-authentication --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.7.2/concepts-clients.md b/site2/website/versioned_docs/version-2.7.2/concepts-clients.md index b68f76a2d8b08..4040624f7d636 100644 --- a/site2/website/versioned_docs/version-2.7.2/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.7.2/concepts-clients.md @@ -5,12 +5,12 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -38,7 +38,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.7.2/concepts-messaging.md b/site2/website/versioned_docs/version-2.7.2/concepts-messaging.md index 688e9dd58446e..3066679d85216 100644 --- a/site2/website/versioned_docs/version-2.7.2/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.7.2/concepts-messaging.md @@ -21,8 +21,8 @@ Messages are the basic "unit" of Pulsar. The following table lists the component Component | Description :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer who produces the message. If you do not specify a producer name, the default name is used. Sequence ID | Each Pulsar message belongs to an ordered sequence on its topic. The sequence ID of the message is its order in that sequence. @@ -30,7 +30,7 @@ Publish time | The timestamp of when the message is published. The timestamp is Event time | An optional timestamp attached to a message by applications. For example, applications attach a timestamp on when the message is processed. If nothing is set to event time, the value is `0`. TypedMessageBuilder | It is used to construct a message. You can set message properties such as the message key, message value with `TypedMessageBuilder`.
    When you set `TypedMessageBuilder`, set the key as a string. If you set the key as other types, for example, an AVRO object, the key is sent as bytes, and it is difficult to get the AVRO object back on the consumer. -> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol). +> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -107,7 +107,7 @@ Messages are received from [brokers](reference-terminology.md#broker) either syn ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -344,7 +344,7 @@ A [subscription](#subscriptions) can have one or more consumers. When a consumer #### When to use -By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry). +By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry.md). #### How to use @@ -448,7 +448,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -460,7 +460,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -498,7 +498,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -553,7 +553,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -576,12 +576,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication.md). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.7.2/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.7.2/concepts-multi-tenancy.md index be752ccbbf9ca..c4acb21acb743 100644 --- a/site2/website/versioned_docs/version-2.7.2/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.7.2/concepts-multi-tenancy.md @@ -5,7 +5,7 @@ sidebar_label: "Multi Tenancy" original_id: concepts-multi-tenancy --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -21,7 +21,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -29,7 +29,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.7.2/concepts-overview.md b/site2/website/versioned_docs/version-2.7.2/concepts-overview.md index c1262b38daf16..e8a2f4b9d321a 100644 --- a/site2/website/versioned_docs/version-2.7.2/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.7.2/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.7.2/concepts-replication.md b/site2/website/versioned_docs/version-2.7.2/concepts-replication.md index 6e23962e99771..799f0eb4d92c6 100644 --- a/site2/website/versioned_docs/version-2.7.2/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.7.2/concepts-replication.md @@ -5,5 +5,5 @@ sidebar_label: "Geo Replication" original_id: concepts-replication --- -Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that. +Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo.md) in Pulsar enables you to do that. diff --git a/site2/website/versioned_docs/version-2.7.2/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.7.2/concepts-tiered-storage.md index 0b45b0a2ed501..f6988e53a8cd4 100644 --- a/site2/website/versioned_docs/version-2.7.2/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.7.2/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.7.2/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.7.2/concepts-topic-compaction.md index e402c965384fd..34b7ed7fbbd31 100644 --- a/site2/website/versioned_docs/version-2.7.2/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.7.2/concepts-topic-compaction.md @@ -7,7 +7,7 @@ original_id: concepts-topic-compaction Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -24,7 +24,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.7.2/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.7.2/cookbooks-compaction.md index 6fcbc197f9ecb..dfa314727241a 100644 --- a/site2/website/versioned_docs/version-2.7.2/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.7.2/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.7.2/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.7.2/cookbooks-non-persistent.md index 391569a8cbc6b..178301e86eb8d 100644 --- a/site2/website/versioned_docs/version-2.7.2/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.7.2/cookbooks-non-persistent.md @@ -41,7 +41,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -55,7 +55,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.7.2/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.7.2/cookbooks-partitioned.md index 39de6de952689..f279f2265f666 100644 --- a/site2/website/versioned_docs/version-2.7.2/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.7.2/cookbooks-partitioned.md @@ -4,4 +4,4 @@ title: Partitioned topics sidebar_label: "Partitioned Topics" original_id: cookbooks-partitioned --- -For details of the content, refer to [manage topics](admin-api-topics). +For details of the content, refer to [manage topics](admin-api-topics.md). diff --git a/site2/website/versioned_docs/version-2.7.2/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.7.2/cookbooks-retention-expiry.md index 5295b4c5de4ae..23175d607b37e 100644 --- a/site2/website/versioned_docs/version-2.7.2/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.7.2/cookbooks-retention-expiry.md @@ -22,7 +22,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.7.2/deploy-aws.md b/site2/website/versioned_docs/version-2.7.2/deploy-aws.md index 1c8f1a08d7249..662e99ceb9b09 100644 --- a/site2/website/versioned_docs/version-2.7.2/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.7.2/deploy-aws.md @@ -5,7 +5,7 @@ sidebar_label: "Amazon Web Services" original_id: deploy-aws --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -210,7 +210,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.7.2/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.7.2/deploy-bare-metal-multi-cluster.md index f8862b3f69e88..0c25f34720800 100644 --- a/site2/website/versioned_docs/version-2.7.2/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.7.2/deploy-bare-metal-multi-cluster.md @@ -9,17 +9,17 @@ original_id: deploy-bare-metal-multi-cluster 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using it in a startup or on a single team, you had better opt for a single cluster. For instructions on deploying a single cluster, -see the guide [here](deploy-bare-metal). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo). Deploying a multi-cluster Pulsar instance involves the following basic steps: +A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md). Deploying a multi-cluster Pulsar instance involves the following basic steps: * Deploying two separate [ZooKeeper](#deploy-zookeeper) quorums: a [local](#deploy-local-zookeeper) quorum for each cluster in the instance and a [configuration store](#configuration-store) quorum for instance-wide tasks * Initializing [cluster metadata](#cluster-metadata-initialization) for each cluster @@ -29,7 +29,7 @@ A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You If you want to deploy a single Pulsar cluster, see [Clusters and Brokers](getting-started-standalone.md#start-the-cluster). > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). ## System requirement @@ -74,7 +74,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -252,7 +252,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -371,17 +371,17 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). -You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > #### Service discovery already provided by many scheduling systems -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ### Service discovery setup -The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol) of Pulsar. +The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol.md) of Pulsar. To get started setting up the built-in service of discovery of Pulsar, you need to change a few parameters in the [`conf/discovery.conf`](reference-configuration.md#service-discovery) configuration file. Set the [`zookeeperServers`](reference-configuration.md#service-discovery-zookeeperServers) parameter to the ZooKeeper quorum connection string of the cluster and the [`configurationStoreServers`](reference-configuration.md#service-discovery-configurationStoreServers) setting to the [configuration store](reference-terminology.md#configuration-store) quorum connection string. @@ -406,7 +406,7 @@ $ bin/pulsar-daemon start discovery ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.7.2/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.7.2/deploy-bare-metal.md index 43f307d9446a9..962476d5c1658 100644 --- a/site2/website/versioned_docs/version-2.7.2/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.7.2/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using Pulsar in a startup or on a single team, it is simplest to opt for a single cluster. If you do need to run a multi-cluster Pulsar instance, -see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -107,7 +107,7 @@ The extracted directory contains the following subdirectories: Directory | Contains :---------|:-------- -`bin` |[command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` |[command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `data` | The data storage directory that ZooKeeper and BookKeeper use `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -269,9 +269,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (you had better not use a different port). -`--web-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). +`--web-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should not use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (you had better not use a different port). -`--broker-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). +`--broker-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). > If you do not have a DNS server, you can use multi-host format in the service URL with the following settings: @@ -411,7 +411,7 @@ webServicePortTls=8443 ### Enable Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -429,7 +429,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker). +If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker.md). ### Start Brokers diff --git a/site2/website/versioned_docs/version-2.7.2/deploy-dcos.md b/site2/website/versioned_docs/version-2.7.2/deploy-dcos.md index f5f8d1faa7dbe..952d5f47e30fa 100644 --- a/site2/website/versioned_docs/version-2.7.2/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.7.2/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -If you want to enable all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of +If you want to enable all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.7.2/deploy-docker.md b/site2/website/versioned_docs/version-2.7.2/deploy-docker.md index 64a0939aa787b..8348d78deb237 100644 --- a/site2/website/versioned_docs/version-2.7.2/deploy-docker.md +++ b/site2/website/versioned_docs/version-2.7.2/deploy-docker.md @@ -57,4 +57,4 @@ docker network connect pulsar broker To check whether the containers are successfully connected to the network, enter the `docker network inspect pulsar` command. -For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal). +For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal.md). diff --git a/site2/website/versioned_docs/version-2.7.2/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.7.2/deploy-kubernetes.md index 248d840c3016d..12b92350e03e8 100644 --- a/site2/website/versioned_docs/version-2.7.2/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.7.2/deploy-kubernetes.md @@ -6,6 +6,6 @@ original_id: deploy-kubernetes --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). diff --git a/site2/website/versioned_docs/version-2.7.2/deploy-monitoring.md b/site2/website/versioned_docs/version-2.7.2/deploy-monitoring.md index de01d1381a207..b0bef95677bc2 100644 --- a/site2/website/versioned_docs/version-2.7.2/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.7.2/deploy-monitoring.md @@ -87,7 +87,7 @@ Those metrics are added in the Prometheus interface, you can monitor and check t You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy). +When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy.md). ## Dashboards @@ -95,7 +95,7 @@ When you collect time series statistics, the major problem is to make sure the n ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager). +The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.7.2/develop-schema.md b/site2/website/versioned_docs/version-2.7.2/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.7.2/develop-schema.md +++ b/site2/website/versioned_docs/version-2.7.2/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.7.2/functions-deploy.md b/site2/website/versioned_docs/version-2.7.2/functions-deploy.md index 51f1140693dbe..d9496b3ed5b48 100644 --- a/site2/website/versioned_docs/version-2.7.2/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.7.2/functions-deploy.md @@ -9,12 +9,12 @@ original_id: functions-deploy To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface diff --git a/site2/website/versioned_docs/version-2.7.2/functions-develop.md b/site2/website/versioned_docs/version-2.7.2/functions-develop.md index 8c7e828153979..0911074c5381f 100644 --- a/site2/website/versioned_docs/version-2.7.2/functions-develop.md +++ b/site2/website/versioned_docs/version-2.7.2/functions-develop.md @@ -1163,7 +1163,7 @@ public class MetricRecorderFunction implements Function { ``` -> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring) guide. +> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring.md) guide.     @@ -1194,11 +1194,11 @@ Currently, the feature is not available in Go. ```` ### Access metrics -To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring) in Pulsar. +To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring.md) in Pulsar. ## Security -If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker). For more details, refer to [Security settings](functions-worker.md#security-settings). +If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings). Pulsar Functions can support the following providers: diff --git a/site2/website/versioned_docs/version-2.7.2/functions-overview.md b/site2/website/versioned_docs/version-2.7.2/functions-overview.md index 25dc6029acf70..816d301e0fd0e 100644 --- a/site2/website/versioned_docs/version-2.7.2/functions-overview.md +++ b/site2/website/versioned_docs/version-2.7.2/functions-overview.md @@ -164,7 +164,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.7.2/functions-package.md b/site2/website/versioned_docs/version-2.7.2/functions-package.md index d7a5b3ec20b60..ac25cb255e110 100644 --- a/site2/website/versioned_docs/version-2.7.2/functions-package.md +++ b/site2/website/versioned_docs/version-2.7.2/functions-package.md @@ -15,7 +15,7 @@ Currently, the window function is not available in Python and Go. ## Prerequisite -Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm). +Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm.md). To check whether the Docker image starts, you can use the `docker ps` command. diff --git a/site2/website/versioned_docs/version-2.7.2/functions-worker.md b/site2/website/versioned_docs/version-2.7.2/functions-worker.md index fa1467aa5fd30..9b603e88de609 100644 --- a/site2/website/versioned_docs/version-2.7.2/functions-worker.md +++ b/site2/website/versioned_docs/version-2.7.2/functions-worker.md @@ -4,7 +4,7 @@ title: Deploy and manage functions worker sidebar_label: "Setup: Pulsar Functions Worker" original_id: functions-worker --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -180,7 +180,7 @@ brokerClientTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -200,7 +200,7 @@ authenticationProviders: [ provider1, provider2 ] ``` For *TLS Authentication* provider, follow the example below to add the necessary settings. -See [TLS Authentication](security-tls-authentication) for more details. +See [TLS Authentication](security-tls-authentication.md) for more details. ``` @@ -224,7 +224,7 @@ properties: ``` For *Token Authentication* provider, add necessary settings for `properties` if needed. -See [Token Authentication](security-jwt) for more details. +See [Token Authentication](security-jwt.md) for more details. ``` @@ -262,7 +262,7 @@ superUserRoles: You can use the public and private key pair that the application configures to perform encryption. Only the consumers with a valid key can decrypt the encrypted messages. -To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption). +To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption.md). We include the relevant configuration information of `CryptoConfig` into `ProducerConfig`. The specific configurable field information about `CryptoConfig` is as follows: diff --git a/site2/website/versioned_docs/version-2.7.2/getting-started-docker.md b/site2/website/versioned_docs/version-2.7.2/getting-started-docker.md index b9f12f0b52688..4f20971d75330 100644 --- a/site2/website/versioned_docs/version-2.7.2/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.7.2/getting-started-docker.md @@ -50,8 +50,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -110,7 +110,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.7.2/getting-started-helm.md b/site2/website/versioned_docs/version-2.7.2/getting-started-helm.md index 65a9fc04fe568..440087c275c05 100644 --- a/site2/website/versioned_docs/version-2.7.2/getting-started-helm.md +++ b/site2/website/versioned_docs/version-2.7.2/getting-started-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.7.2/getting-started-pulsar.md b/site2/website/versioned_docs/version-2.7.2/getting-started-pulsar.md index 11c5e66cb62c2..752590f57b558 100644 --- a/site2/website/versioned_docs/version-2.7.2/getting-started-pulsar.md +++ b/site2/website/versioned_docs/version-2.7.2/getting-started-pulsar.md @@ -5,13 +5,13 @@ sidebar_label: "Pulsar 2.0" original_id: pulsar-2.0 --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.7.2/getting-started-standalone.md b/site2/website/versioned_docs/version-2.7.2/getting-started-standalone.md index 95782fc6ca8e1..05babc33a8bdb 100644 --- a/site2/website/versioned_docs/version-2.7.2/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.7.2/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -65,7 +65,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -74,7 +74,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -123,7 +123,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -137,7 +137,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -170,12 +170,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -208,7 +208,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.7.2/helm-install.md b/site2/website/versioned_docs/version-2.7.2/helm-install.md index edb12260f010e..1f4d5eb69d5dd 100644 --- a/site2/website/versioned_docs/version-2.7.2/helm-install.md +++ b/site2/website/versioned_docs/version-2.7.2/helm-install.md @@ -21,7 +21,7 @@ Before deploying Pulsar, you need to prepare your environment. ### Tools -Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools) on your computer. +Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools.md) on your computer. ## Cloud cluster preparation @@ -37,8 +37,8 @@ To create and connect to the Kubernetes cluster, follow the instructions: ## Pulsar deployment -Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy). +Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy.md). ## Pulsar upgrade -To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade). +To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade.md). diff --git a/site2/website/versioned_docs/version-2.7.2/helm-overview.md b/site2/website/versioned_docs/version-2.7.2/helm-overview.md index 9cde85e2d18d5..385d535e319b6 100644 --- a/site2/website/versioned_docs/version-2.7.2/helm-overview.md +++ b/site2/website/versioned_docs/version-2.7.2/helm-overview.md @@ -52,9 +52,9 @@ It includes support for: ## Pulsar Helm chart quick start -To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install). +This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install.md). ## Troubleshooting @@ -64,7 +64,7 @@ We have done our best to make these charts as seamless as possible. Occasionally The Apache Pulsar Helm chart contains all required dependencies. -If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm) for your first iteration. +If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm.md) for your first iteration. 1. [Preparation](helm-prepare.md) 2. [Deployment](helm-deploy.md) @@ -82,7 +82,7 @@ helm upgrade apache/pulsar -f pulsar.yaml ``` -For more detailed information, see [Upgrading](helm-upgrade). +For more detailed information, see [Upgrading](helm-upgrade.md). ## Uninstallation diff --git a/site2/website/versioned_docs/version-2.7.2/helm-prepare.md b/site2/website/versioned_docs/version-2.7.2/helm-prepare.md index 95d7e668f9eca..5e9f2f9ef4f68 100644 --- a/site2/website/versioned_docs/version-2.7.2/helm-prepare.md +++ b/site2/website/versioned_docs/version-2.7.2/helm-prepare.md @@ -89,4 +89,4 @@ PROJECT= USE_LOCAL_SSD=true LOCAL_SSD_COUNT= ## Next Steps -Continue with the [installation of the chart](helm-deploy) once you have the cluster up and running. +Continue with the [installation of the chart](helm-deploy.md) once you have the cluster up and running. diff --git a/site2/website/versioned_docs/version-2.7.2/helm-tools.md b/site2/website/versioned_docs/version-2.7.2/helm-tools.md index 6894608b03473..a7e4017b4fc4f 100644 --- a/site2/website/versioned_docs/version-2.7.2/helm-tools.md +++ b/site2/website/versioned_docs/version-2.7.2/helm-tools.md @@ -25,7 +25,7 @@ You can get Helm from the project's [releases page](https://github.com/helm/helm ### Next steps -Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare). +Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare.md). ## Additional information diff --git a/site2/website/versioned_docs/version-2.7.2/helm-upgrade.md b/site2/website/versioned_docs/version-2.7.2/helm-upgrade.md index e39cf97407355..7d671e6bfb3c1 100644 --- a/site2/website/versioned_docs/version-2.7.2/helm-upgrade.md +++ b/site2/website/versioned_docs/version-2.7.2/helm-upgrade.md @@ -20,7 +20,7 @@ You can retrieve your previous `--set` arguments cleanly, with `helm get values To upgrade Apache Pulsar to a newer version, follow these steps: 1. Check the change log for the specific version you would like to upgrade to. -2. Go through [deployment documentation](helm-deploy) step by step. +2. Go through [deployment documentation](helm-deploy.md) step by step. 3. Extract your previous `--set` arguments with the following command. ```bash diff --git a/site2/website/versioned_docs/version-2.7.2/io-cdc.md b/site2/website/versioned_docs/version-2.7.2/io-cdc.md index 20f16ae0d6904..e6e662884826d 100644 --- a/site2/website/versioned_docs/version-2.7.2/io-cdc.md +++ b/site2/website/versioned_docs/version-2.7.2/io-cdc.md @@ -13,8 +13,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.7.2/io-develop.md b/site2/website/versioned_docs/version-2.7.2/io-develop.md index 4a815b3b9d6f6..7a356662560c9 100644 --- a/site2/website/versioned_docs/version-2.7.2/io-develop.md +++ b/site2/website/versioned_docs/version-2.7.2/io-develop.md @@ -8,15 +8,15 @@ original_id: io-develop This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -163,7 +163,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -192,7 +192,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). diff --git a/site2/website/versioned_docs/version-2.7.2/io-overview.md b/site2/website/versioned_docs/version-2.7.2/io-overview.md index 68df2472d6334..b538604edc46f 100644 --- a/site2/website/versioned_docs/version-2.7.2/io-overview.md +++ b/site2/website/versioned_docs/version-2.7.2/io-overview.md @@ -158,7 +158,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.7.2/io-quickstart.md b/site2/website/versioned_docs/version-2.7.2/io-quickstart.md index b8fea1560bdeb..8474c93f51336 100644 --- a/site2/website/versioned_docs/version-2.7.2/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.7.2/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you are able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you are able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However, all the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -109,7 +109,7 @@ This section demonstrates how to connect Pulsar to Cassandra. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink.md). ::: @@ -236,11 +236,11 @@ You can create a configuration file through one of the following methods. ``` -For more information, see [Cassandra sink connector](io-cassandra-sink). +For more information, see [Cassandra sink connector](io-cassandra-sink.md). ### Create a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on them. Run the following command to create a Cassandra sink connector with sink type _cassandra_ and the config file _examples/cassandra-sink.yml_ created previously. @@ -267,7 +267,7 @@ as a Pulsar Function and writes the messages produced in the topic _test_cassand ### Inspect a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * Get the information of a Cassandra sink. @@ -431,7 +431,7 @@ to monitor a connector and perform other operations on it. ### Delete a Cassandra Sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. ```bash @@ -454,7 +454,7 @@ This section demonstrates how to connect Pulsar to PostgreSQL. ::: ->For more information, see [JDBC sink connector](io-jdbc-sink). +>For more information, see [JDBC sink connector](io-jdbc-sink.md). ### Setup a PostgreSQL cluster @@ -614,7 +614,7 @@ In this section, you need to configure a JDBC sink connector. ### Create a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on it. This example creates a sink connector and specifies the desired information. @@ -660,7 +660,7 @@ The sink has been created successfully if the following message appears. ### Inspect a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * List all running JDBC sink(s). @@ -780,7 +780,7 @@ to monitor a connector and perform other operations on it. ### Stop a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to stop a connector and perform other operations on it. ```bash @@ -808,7 +808,7 @@ The sink instance has been stopped successfully if the following message disappe ### Restart a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to restart a connector and perform other operations on it. ```bash @@ -844,7 +844,7 @@ Note that `pulsar-admin sinks localrun options` **runs a sink connector locally* ### Update a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to update a connector and perform other operations on it. This example updates the parallelism of the _pulsar-postgres-jdbc-sink_ sink connector to 2. @@ -912,7 +912,7 @@ The result shows that the parallelism is 2. ### Delete a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. This example deletes the _pulsar-postgres-jdbc-sink_ sink connector. diff --git a/site2/website/versioned_docs/version-2.7.2/io-use.md b/site2/website/versioned_docs/version-2.7.2/io-use.md index b6409d4420ad7..da9ed746c4d37 100644 --- a/site2/website/versioned_docs/version-2.7.2/io-use.md +++ b/site2/website/versioned_docs/version-2.7.2/io-use.md @@ -15,7 +15,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.7.2/kubernetes-helm.md b/site2/website/versioned_docs/version-2.7.2/kubernetes-helm.md index a9f768e6a96b3..ea92a0968cd7d 100644 --- a/site2/website/versioned_docs/version-2.7.2/kubernetes-helm.md +++ b/site2/website/versioned_docs/version-2.7.2/kubernetes-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.7.2/reference-cli-tools.md b/site2/website/versioned_docs/version-2.7.2/reference-cli-tools.md index d17bd3efd9790..2bc3cc56130f5 100644 --- a/site2/website/versioned_docs/version-2.7.2/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.7.2/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.7.2/schema-get-started.md b/site2/website/versioned_docs/version-2.7.2/schema-get-started.md index 97e270586a26d..afacb0fa51f2e 100644 --- a/site2/website/versioned_docs/version-2.7.2/schema-get-started.md +++ b/site2/website/versioned_docs/version-2.7.2/schema-get-started.md @@ -17,7 +17,7 @@ Applications typically adopt one of the following approaches to guarantee type s #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ### Client-side approach diff --git a/site2/website/versioned_docs/version-2.7.2/schema-manage.md b/site2/website/versioned_docs/version-2.7.2/schema-manage.md index 98105408fb37d..5984fdbeaaa5f 100644 --- a/site2/website/versioned_docs/version-2.7.2/schema-manage.md +++ b/site2/website/versioned_docs/version-2.7.2/schema-manage.md @@ -160,7 +160,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.7.2/schema-understand.md b/site2/website/versioned_docs/version-2.7.2/schema-understand.md index 333c902b58319..d17c6168da59f 100644 --- a/site2/website/versioned_docs/version-2.7.2/schema-understand.md +++ b/site2/website/versioned_docs/version-2.7.2/schema-understand.md @@ -380,7 +380,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```java diff --git a/site2/website/versioned_docs/version-2.7.2/security-athenz.md b/site2/website/versioned_docs/version-2.7.2/security-athenz.md index 947c3f470be46..8a39fe25316d0 100644 --- a/site2/website/versioned_docs/version-2.7.2/security-athenz.md +++ b/site2/website/versioned_docs/version-2.7.2/security-athenz.md @@ -76,7 +76,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.7.2/security-authorization.md b/site2/website/versioned_docs/version-2.7.2/security-authorization.md index e6785873f0e10..7ac09b6f439ea 100644 --- a/site2/website/versioned_docs/version-2.7.2/security-authorization.md +++ b/site2/website/versioned_docs/version-2.7.2/security-authorization.md @@ -27,7 +27,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -59,7 +59,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -87,7 +87,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication diff --git a/site2/website/versioned_docs/version-2.7.2/security-bouncy-castle.md b/site2/website/versioned_docs/version-2.7.2/security-bouncy-castle.md index abc25c71b8ae2..be937055d8e31 100644 --- a/site2/website/versioned_docs/version-2.7.2/security-bouncy-castle.md +++ b/site2/website/versioned_docs/version-2.7.2/security-bouncy-castle.md @@ -16,7 +16,7 @@ In Pulsar, security and crypto have dependencies on BouncyCastle Jars. For the d `Bouncy Castle` provides both [FIPS](https://www.bouncycastle.org/fips_faq.html) and non-FIPS version. But in a JVM, you can not include both of the 2 versions, and you need to exclude the current version before include the other. -In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. +In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption.md). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. ## How BouncyCastle modules packaged in Pulsar diff --git a/site2/website/versioned_docs/version-2.7.2/security-extending.md b/site2/website/versioned_docs/version-2.7.2/security-extending.md index c088e3a2c1c2c..e7484453b8beb 100644 --- a/site2/website/versioned_docs/version-2.7.2/security-extending.md +++ b/site2/website/versioned_docs/version-2.7.2/security-extending.md @@ -9,7 +9,7 @@ Pulsar provides a way to use custom authentication and authorization mechanisms. ## Authentication -Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview). +Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview.md). You can use a custom authentication mechanism by providing the implementation in the form of two plugins. One plugin is for the Client library and the other plugin is for the Pulsar Proxy and/or Pulsar Broker to validate the credentials. diff --git a/site2/website/versioned_docs/version-2.7.2/security-jwt.md b/site2/website/versioned_docs/version-2.7.2/security-jwt.md index 656ab362f71b0..91117bedff753 100644 --- a/site2/website/versioned_docs/version-2.7.2/security-jwt.md +++ b/site2/website/versioned_docs/version-2.7.2/security-jwt.md @@ -32,11 +32,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -303,7 +303,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.7.2/security-kerberos.md b/site2/website/versioned_docs/version-2.7.2/security-kerberos.md index 670586a9d1691..c49fa3bea1fce 100644 --- a/site2/website/versioned_docs/version-2.7.2/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.7.2/security-kerberos.md @@ -373,7 +373,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.7.2/security-overview.md b/site2/website/versioned_docs/version-2.7.2/security-overview.md index c7823104d145b..a1664fcb8b450 100644 --- a/site2/website/versioned_docs/version-2.7.2/security-overview.md +++ b/site2/website/versioned_docs/version-2.7.2/security-overview.md @@ -21,7 +21,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.7.2/security-tls-authentication.md b/site2/website/versioned_docs/version-2.7.2/security-tls-authentication.md index 79aa8ea9fd13d..61c8aa43e1421 100644 --- a/site2/website/versioned_docs/version-2.7.2/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.7.2/security-tls-authentication.md @@ -7,9 +7,9 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -101,7 +101,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -121,7 +121,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.7.2/security-tls-keystore.md b/site2/website/versioned_docs/version-2.7.2/security-tls-keystore.md index 062d903e0e2ca..cb14afd1e4080 100644 --- a/site2/website/versioned_docs/version-2.7.2/security-tls-keystore.md +++ b/site2/website/versioned_docs/version-2.7.2/security-tls-keystore.md @@ -7,7 +7,7 @@ original_id: security-tls-keystore ## Overview -Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication) between clients and Apache Pulsar service. +Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication.md) between clients and Apache Pulsar service. By default it uses PEM format file configuration. This page tries to describe use [KeyStore](https://en.wikipedia.org/wiki/Java_KeyStore) type configure for TLS. @@ -165,7 +165,7 @@ This is similar to [TLS encryption configuing for client with PEM type](security For a a minimal configuration, user need to provide the TrustStore information. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties @@ -258,7 +258,7 @@ webSocketServiceEnabled=false Besides the TLS encryption configuring. The main work is configuring the KeyStore, which contains a valid CN as client role, for client. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties diff --git a/site2/website/versioned_docs/version-2.7.2/security-tls-transport.md b/site2/website/versioned_docs/version-2.7.2/security-tls-transport.md index 083aca48f65e2..52709e4867032 100644 --- a/site2/website/versioned_docs/version-2.7.2/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.7.2/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,9 +27,9 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). -`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ## Create TLS certificates @@ -130,7 +130,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Broker Configuration -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -201,7 +201,7 @@ The examples below show that hostname verification is disabled for the CLI tools ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.7.2/security-token-admin.md b/site2/website/versioned_docs/version-2.7.2/security-token-admin.md index 2173a6cb1d11d..a5d69da9f0602 100644 --- a/site2/website/versioned_docs/version-2.7.2/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.7.2/security-token-admin.md @@ -163,7 +163,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.7.2/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.7.2/sql-deployment-configurations.md index c6d7c88a07e94..cc8bdbcd4fd51 100644 --- a/site2/website/versioned_docs/version-2.7.2/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.7.2/sql-deployment-configurations.md @@ -64,7 +64,7 @@ Since Pulsar SQL is powered by [Presto](https://prestosql.io), the configuration :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.7.2/sql-overview.md b/site2/website/versioned_docs/version-2.7.2/sql-overview.md index 3303923c6d237..e4dee1284869b 100644 --- a/site2/website/versioned_docs/version-2.7.2/sql-overview.md +++ b/site2/website/versioned_docs/version-2.7.2/sql-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Overview" original_id: sql-overview --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.7.2/standalone-docker.md b/site2/website/versioned_docs/version-2.7.2/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.7.2/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.7.2/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.7.2/standalone.md b/site2/website/versioned_docs/version-2.7.2/standalone.md index 2ff2d6059a284..292b344effe6e 100644 --- a/site2/website/versioned_docs/version-2.7.2/standalone.md +++ b/site2/website/versioned_docs/version-2.7.2/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -65,7 +65,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -74,7 +74,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -123,7 +123,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -137,7 +137,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -170,12 +170,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -208,7 +208,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.7.2/tiered-storage-overview.md b/site2/website/versioned_docs/version-2.7.2/tiered-storage-overview.md index 7c6528ce2b36c..c635034f463b4 100644 --- a/site2/website/versioned_docs/version-2.7.2/tiered-storage-overview.md +++ b/site2/website/versioned_docs/version-2.7.2/tiered-storage-overview.md @@ -13,9 +13,9 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws). + - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws.md). - - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs). + - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs.md). ::: @@ -25,7 +25,7 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem). + For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem.md). ::: diff --git a/site2/website/versioned_docs/version-2.7.3/admin-api-brokers.md b/site2/website/versioned_docs/version-2.7.3/admin-api-brokers.md index 751bc14983205..61047c4d631d1 100644 --- a/site2/website/versioned_docs/version-2.7.3/admin-api-brokers.md +++ b/site2/website/versioned_docs/version-2.7.3/admin-api-brokers.md @@ -18,7 +18,7 @@ Pulsar brokers consist of two components: [Brokers](reference-terminology.md#broker) can be managed via: -* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/brokers` endpoint of the admin {@inject: rest:REST:/} API * The `brokers` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin.html} object in the [Java API](client-libraries-java.md) @@ -123,7 +123,7 @@ One way to configure a Pulsar [broker](reference-terminology.md#broker) is to su But since all broker configuration in Pulsar is stored in ZooKeeper, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values. -* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. +* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. * In the Pulsar admin {@inject: rest:REST:/} API, dynamic configuration is managed through the `/admin/v2/brokers/configuration` endpoint. ### Update dynamic configuration diff --git a/site2/website/versioned_docs/version-2.7.3/admin-api-clusters.md b/site2/website/versioned_docs/version-2.7.3/admin-api-clusters.md index f3af175295514..79508763e1db0 100644 --- a/site2/website/versioned_docs/version-2.7.3/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.7.3/admin-api-clusters.md @@ -16,7 +16,7 @@ servers (aka [bookies](reference-terminology.md#bookie)), and a [ZooKeeper](http Clusters can be managed via: -* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/clusters` endpoint of the admin {@inject: rest:REST:/} API * The `clusters` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md) @@ -79,7 +79,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -103,11 +103,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block -If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: +If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin.md) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: |Name|Description|Default| |----|-----------|-------| diff --git a/site2/website/versioned_docs/version-2.7.3/admin-api-partitioned-topics.md b/site2/website/versioned_docs/version-2.7.3/admin-api-partitioned-topics.md index 28338e8cc4a69..4582ba14772d8 100644 --- a/site2/website/versioned_docs/version-2.7.3/admin-api-partitioned-topics.md +++ b/site2/website/versioned_docs/version-2.7.3/admin-api-partitioned-topics.md @@ -5,4 +5,4 @@ sidebar_label: "Partitioned topics" original_id: admin-api-partitioned-topics --- -For details of the content, refer to [manage topics](admin-api-topics). +For details of the content, refer to [manage topics](admin-api-topics.md). diff --git a/site2/website/versioned_docs/version-2.7.3/admin-api-persistent-topics.md b/site2/website/versioned_docs/version-2.7.3/admin-api-persistent-topics.md index 596e0358227ea..13e24f1d9eba4 100644 --- a/site2/website/versioned_docs/version-2.7.3/admin-api-persistent-topics.md +++ b/site2/website/versioned_docs/version-2.7.3/admin-api-persistent-topics.md @@ -5,4 +5,4 @@ sidebar_label: "Persistent topics" original_id: admin-api-persistent-topics --- -For details of the content, refer to [manage topics](admin-api-topics). +For details of the content, refer to [manage topics](admin-api-topics.md). diff --git a/site2/website/versioned_docs/version-2.7.3/admin-api-tenants.md b/site2/website/versioned_docs/version-2.7.3/admin-api-tenants.md index 63afe58620c62..d78aa2e55f4c3 100644 --- a/site2/website/versioned_docs/version-2.7.3/admin-api-tenants.md +++ b/site2/website/versioned_docs/version-2.7.3/admin-api-tenants.md @@ -11,7 +11,7 @@ import TabItem from '@theme/TabItem'; ```` -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -108,7 +108,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.7.3/client-libraries-cgo.md b/site2/website/versioned_docs/version-2.7.3/client-libraries-cgo.md index c79f7bb965473..f352f942b7714 100644 --- a/site2/website/versioned_docs/version-2.7.3/client-libraries-cgo.md +++ b/site2/website/versioned_docs/version-2.7.3/client-libraries-cgo.md @@ -24,7 +24,7 @@ Currently, the following Go clients are maintained in two repositories. ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Install go package @@ -57,7 +57,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -75,7 +75,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -236,14 +236,14 @@ Parameter | Description | Default `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | `Properties`| Attach a set of application defined properties to the producer. This properties will be visible in the topic stats | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | Set the number of max pending messages across all the partitions. This setting will be used to lower the max pending messages for each partition `MaxPendingMessages(int)`, if the total exceeds the configured value.| `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | `Batching` | Control whether automatic batching of messages is enabled for the producer. | false `BatchingMaxPublishDelay` | Set the time period within which the messages sent will be batched (default: 1ms) if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or until | 1ms `BatchingMaxMessages` | Set the maximum number of messages permitted in a batch. (default: 1000) If set to a value greater than 1, messages will be queued until this threshold is reached or batch interval has elapsed | 1000 @@ -512,7 +512,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.7.3/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.7.3/client-libraries-cpp.md index bb52d9effa404..684df5fd2d825 100644 --- a/site2/website/versioned_docs/version-2.7.3/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.7.3/client-libraries-cpp.md @@ -261,7 +261,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema -This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started). +This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started.md). ### Create producer with Avro schema diff --git a/site2/website/versioned_docs/version-2.7.3/client-libraries-dotnet.md b/site2/website/versioned_docs/version-2.7.3/client-libraries-dotnet.md index 4e0afe3d2ff6b..fbec5e473be69 100644 --- a/site2/website/versioned_docs/version-2.7.3/client-libraries-dotnet.md +++ b/site2/website/versioned_docs/version-2.7.3/client-libraries-dotnet.md @@ -170,7 +170,7 @@ var client = PulsarClient.Builder() Currently, the Pulsar C# client supports the TLS (Transport Layer Security) and JWT (JSON Web Token) authentication. -If you have followed [Authentication using TLS](security-tls-authentication), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: +If you have followed [Authentication using TLS](security-tls-authentication.md), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: 1. Create an unencrypted and password-less pfx file. diff --git a/site2/website/versioned_docs/version-2.7.3/client-libraries-go.md b/site2/website/versioned_docs/version-2.7.3/client-libraries-go.md index 97705a643c049..6c05e1d733422 100644 --- a/site2/website/versioned_docs/version-2.7.3/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.7.3/client-libraries-go.md @@ -35,7 +35,7 @@ import "github.com/apache/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -53,7 +53,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -680,7 +680,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker @@ -700,7 +700,7 @@ opts := pulsar.ClientOptions{ ## OAuth2 authentication -To use [OAuth2 authentication](security-oauth2), you'll need to configure your client to perform the following operations. +To use [OAuth2 authentication](security-oauth2.md), you'll need to configure your client to perform the following operations. This example shows how to configure OAuth2 authentication. ```go diff --git a/site2/website/versioned_docs/version-2.7.3/client-libraries-java.md b/site2/website/versioned_docs/version-2.7.3/client-libraries-java.md index bbe9473622fbe..ed7ce010e7550 100644 --- a/site2/website/versioned_docs/version-2.7.3/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.7.3/client-libraries-java.md @@ -5,7 +5,7 @@ sidebar_label: "Java" original_id: client-libraries-java --- -You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), and [reader](#reader) of a Java client are thread-safe. @@ -14,10 +14,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Includes both `pulsar-client` and `pulsar-client-admin`

    Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.

    In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -57,7 +57,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -83,7 +83,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -114,7 +114,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -237,7 +237,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -822,7 +822,7 @@ Total hash range size is 65536, so the max end of the range should be less than ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -936,7 +936,7 @@ The following schema formats are currently available for Java: ## Authentication -Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2). You can use the Pulsar Java client with all of them. +Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2.md). You can use the Pulsar Java client with all of them. ### TLS Authentication @@ -963,7 +963,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` @@ -1000,7 +1000,7 @@ PulsarClient client = PulsarClient.builder() ### Oauth2 -The following example shows how to use [Oauth2](security-oauth2) as an authentication provider for the Pulsar Java client. +The following example shows how to use [Oauth2](security-oauth2.md) as an authentication provider for the Pulsar Java client. You can use the factory method to configure authentication for Pulsar Java client. diff --git a/site2/website/versioned_docs/version-2.7.3/client-libraries-node.md b/site2/website/versioned_docs/version-2.7.3/client-libraries-node.md index 12c1bdf590f1a..db851088a571f 100644 --- a/site2/website/versioned_docs/version-2.7.3/client-libraries-node.md +++ b/site2/website/versioned_docs/version-2.7.3/client-libraries-node.md @@ -46,7 +46,7 @@ Also, this library works only in Node.js 10.x or later because it uses the [`nod ::: ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here is an example for `localhost`: @@ -64,7 +64,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication), the URL will look like something like this: +If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication.md), the URL will look like something like this: ```http @@ -99,7 +99,7 @@ The following configurable parameters are available for Pulsar clients: | Parameter | Description | Default | | :-------- | :---------- | :------ | | `serviceUrl` | The connection URL for the Pulsar cluster. See [above](#connection-urls) for more info. | | -| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication) for more info. | | +| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication.md) for more info. | | | `operationTimeoutSeconds` | The timeout for Node.js client operations (creating producers, subscribing to and unsubscribing from [topics](reference-terminology.md#topic)). Retries will occur until this threshold is reached, at which point the operation will fail. | 30 | | `ioThreads` | The number of threads to use for handling connections to Pulsar [brokers](reference-terminology.md#broker). | 1 | | `messageListenerThreads` | The number of threads used by message listeners ([consumers](#consumers) and [readers](#readers)). | 1 | @@ -150,7 +150,7 @@ Pulsar Node.js producers have the following methods available: | :-------- | :---------- | :------ | | `topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer publishes messages. The topic format is `` or `//`. For example, `sample/ns1/my-topic`. | | | `producerName` | A name for the producer. If you do not explicitly assign a name, Pulsar will automatically generate a globally unique name. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | | -| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30000 | +| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30000 | | `initialSequenceId` | The initial sequence ID of the message. When producer send message, add sequence ID to message. The ID is increased each time to send. | | | `maxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `send` method will fail *unless* `blockIfQueueFull` is set to `true`. | 1000 | | `maxPendingMessagesAcrossPartitions` | The maximum size of the sum of partition's pending queue. | 50000 | diff --git a/site2/website/versioned_docs/version-2.7.3/client-libraries-python.md b/site2/website/versioned_docs/version-2.7.3/client-libraries-python.md index 7d44db909ca9f..e7ea7f01813f8 100644 --- a/site2/website/versioned_docs/version-2.7.3/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.7.3/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. diff --git a/site2/website/versioned_docs/version-2.7.3/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.7.3/client-libraries-websocket.md index cdf87d70aa905..4906a93ec0e6f 100644 --- a/site2/website/versioned_docs/version-2.7.3/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.7.3/client-libraries-websocket.md @@ -5,14 +5,14 @@ sidebar_label: "WebSocket" original_id: client-libraries-websocket --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.7.3/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.7.3/concepts-architecture-overview.md index c65f9149ceda5..ad3f04858b25a 100644 --- a/site2/website/versioned_docs/version-2.7.3/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.7.3/concepts-architecture-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Architecture" original_id: concepts-architecture-overview --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -17,20 +17,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -40,9 +40,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -130,17 +130,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -148,7 +148,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.7.3/concepts-authentication.md b/site2/website/versioned_docs/version-2.7.3/concepts-authentication.md index b375ecb16a2d8..f6307890c904a 100644 --- a/site2/website/versioned_docs/version-2.7.3/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.7.3/concepts-authentication.md @@ -5,5 +5,5 @@ sidebar_label: "Authentication and Authorization" original_id: concepts-authentication --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.7.3/concepts-clients.md b/site2/website/versioned_docs/version-2.7.3/concepts-clients.md index b68f76a2d8b08..4040624f7d636 100644 --- a/site2/website/versioned_docs/version-2.7.3/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.7.3/concepts-clients.md @@ -5,12 +5,12 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -38,7 +38,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.7.3/concepts-messaging.md b/site2/website/versioned_docs/version-2.7.3/concepts-messaging.md index a4ea0637c94eb..4b03824755273 100644 --- a/site2/website/versioned_docs/version-2.7.3/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.7.3/concepts-messaging.md @@ -21,8 +21,8 @@ Messages are the basic "unit" of Pulsar. The following table lists the component Component | Description :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer who produces the message. If you do not specify a producer name, the default name is used. Sequence ID | Each Pulsar message belongs to an ordered sequence on its topic. The sequence ID of the message is its order in that sequence. @@ -30,7 +30,7 @@ Publish time | The timestamp of when the message is published. The timestamp is Event time | An optional timestamp attached to a message by applications. For example, applications attach a timestamp on when the message is processed. If nothing is set to event time, the value is `0`. TypedMessageBuilder | It is used to construct a message. You can set message properties such as the message key, message value with `TypedMessageBuilder`.
    When you set `TypedMessageBuilder`, set the key as a string. If you set the key as other types, for example, an AVRO object, the key is sent as bytes, and it is difficult to get the AVRO object back on the consumer. -> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol). +> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -107,7 +107,7 @@ Messages are received from [brokers](reference-terminology.md#broker) either syn ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -344,7 +344,7 @@ A [subscription](#subscriptions) can have one or more consumers. When a consumer #### When to use -By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry). +By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry.md). #### How to use @@ -448,7 +448,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -460,7 +460,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -498,7 +498,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -553,7 +553,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -576,12 +576,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication.md). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.7.3/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.7.3/concepts-multi-tenancy.md index be752ccbbf9ca..c4acb21acb743 100644 --- a/site2/website/versioned_docs/version-2.7.3/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.7.3/concepts-multi-tenancy.md @@ -5,7 +5,7 @@ sidebar_label: "Multi Tenancy" original_id: concepts-multi-tenancy --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -21,7 +21,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -29,7 +29,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.7.3/concepts-overview.md b/site2/website/versioned_docs/version-2.7.3/concepts-overview.md index c1262b38daf16..e8a2f4b9d321a 100644 --- a/site2/website/versioned_docs/version-2.7.3/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.7.3/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.7.3/concepts-replication.md b/site2/website/versioned_docs/version-2.7.3/concepts-replication.md index 6e23962e99771..799f0eb4d92c6 100644 --- a/site2/website/versioned_docs/version-2.7.3/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.7.3/concepts-replication.md @@ -5,5 +5,5 @@ sidebar_label: "Geo Replication" original_id: concepts-replication --- -Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that. +Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo.md) in Pulsar enables you to do that. diff --git a/site2/website/versioned_docs/version-2.7.3/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.7.3/concepts-tiered-storage.md index 0b45b0a2ed501..f6988e53a8cd4 100644 --- a/site2/website/versioned_docs/version-2.7.3/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.7.3/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.7.3/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.7.3/concepts-topic-compaction.md index e402c965384fd..34b7ed7fbbd31 100644 --- a/site2/website/versioned_docs/version-2.7.3/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.7.3/concepts-topic-compaction.md @@ -7,7 +7,7 @@ original_id: concepts-topic-compaction Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -24,7 +24,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.7.3/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.7.3/cookbooks-compaction.md index 6fcbc197f9ecb..dfa314727241a 100644 --- a/site2/website/versioned_docs/version-2.7.3/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.7.3/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.7.3/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.7.3/cookbooks-non-persistent.md index 391569a8cbc6b..178301e86eb8d 100644 --- a/site2/website/versioned_docs/version-2.7.3/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.7.3/cookbooks-non-persistent.md @@ -41,7 +41,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -55,7 +55,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.7.3/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.7.3/cookbooks-partitioned.md index 39de6de952689..f279f2265f666 100644 --- a/site2/website/versioned_docs/version-2.7.3/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.7.3/cookbooks-partitioned.md @@ -4,4 +4,4 @@ title: Partitioned topics sidebar_label: "Partitioned Topics" original_id: cookbooks-partitioned --- -For details of the content, refer to [manage topics](admin-api-topics). +For details of the content, refer to [manage topics](admin-api-topics.md). diff --git a/site2/website/versioned_docs/version-2.7.3/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.7.3/cookbooks-retention-expiry.md index 5295b4c5de4ae..23175d607b37e 100644 --- a/site2/website/versioned_docs/version-2.7.3/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.7.3/cookbooks-retention-expiry.md @@ -22,7 +22,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.7.3/deploy-aws.md b/site2/website/versioned_docs/version-2.7.3/deploy-aws.md index 1c8f1a08d7249..662e99ceb9b09 100644 --- a/site2/website/versioned_docs/version-2.7.3/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.7.3/deploy-aws.md @@ -5,7 +5,7 @@ sidebar_label: "Amazon Web Services" original_id: deploy-aws --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -210,7 +210,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.7.3/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.7.3/deploy-bare-metal-multi-cluster.md index 4ee6da37f0a90..74303a3ff7924 100644 --- a/site2/website/versioned_docs/version-2.7.3/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.7.3/deploy-bare-metal-multi-cluster.md @@ -9,17 +9,17 @@ original_id: deploy-bare-metal-multi-cluster 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using it in a startup or on a single team, you had better opt for a single cluster. For instructions on deploying a single cluster, -see the guide [here](deploy-bare-metal). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo). Deploying a multi-cluster Pulsar instance involves the following basic steps: +A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md). Deploying a multi-cluster Pulsar instance involves the following basic steps: * Deploying two separate [ZooKeeper](#deploy-zookeeper) quorums: a [local](#deploy-local-zookeeper) quorum for each cluster in the instance and a [configuration store](#configuration-store) quorum for instance-wide tasks * Initializing [cluster metadata](#cluster-metadata-initialization) for each cluster @@ -29,7 +29,7 @@ A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You If you want to deploy a single Pulsar cluster, see [Clusters and Brokers](getting-started-standalone.md#start-the-cluster). > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). ## System requirement Pulsar is currently available for **MacOS** and **Linux**. In order to use Pulsar, you need to install Java 8 from [Oracle download center](http://www.oracle.com/). @@ -67,7 +67,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -245,7 +245,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -364,17 +364,17 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). -You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > #### Service discovery already provided by many scheduling systems -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ### Service discovery setup -The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol) of Pulsar. +The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol.md) of Pulsar. To get started setting up the built-in service of discovery of Pulsar, you need to change a few parameters in the [`conf/discovery.conf`](reference-configuration.md#service-discovery) configuration file. Set the [`zookeeperServers`](reference-configuration.md#service-discovery-zookeeperServers) parameter to the ZooKeeper quorum connection string of the cluster and the [`configurationStoreServers`](reference-configuration.md#service-discovery-configurationStoreServers) setting to the [configuration store](reference-terminology.md#configuration-store) quorum connection string. @@ -399,7 +399,7 @@ $ bin/pulsar-daemon start discovery ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.7.3/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.7.3/deploy-bare-metal.md index 7b1850b3b21aa..1ea6286f6e533 100644 --- a/site2/website/versioned_docs/version-2.7.3/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.7.3/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using Pulsar in a startup or on a single team, it is simplest to opt for a single cluster. If you do need to run a multi-cluster Pulsar instance, -see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -99,7 +99,7 @@ The extracted directory contains the following subdirectories: Directory | Contains :---------|:-------- -`bin` |[command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` |[command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `data` | The data storage directory that ZooKeeper and BookKeeper use `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -261,9 +261,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (you had better not use a different port). -`--web-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). +`--web-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should not use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (you had better not use a different port). -`--broker-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). +`--broker-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). > If you do not have a DNS server, you can use multi-host format in the service URL with the following settings: @@ -403,7 +403,7 @@ webServicePortTls=8443 ### Enable Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -421,7 +421,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker). +If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker.md). ### Start Brokers diff --git a/site2/website/versioned_docs/version-2.7.3/deploy-dcos.md b/site2/website/versioned_docs/version-2.7.3/deploy-dcos.md index f5f8d1faa7dbe..952d5f47e30fa 100644 --- a/site2/website/versioned_docs/version-2.7.3/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.7.3/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -If you want to enable all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of +If you want to enable all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.7.3/deploy-docker.md b/site2/website/versioned_docs/version-2.7.3/deploy-docker.md index 64a0939aa787b..8348d78deb237 100644 --- a/site2/website/versioned_docs/version-2.7.3/deploy-docker.md +++ b/site2/website/versioned_docs/version-2.7.3/deploy-docker.md @@ -57,4 +57,4 @@ docker network connect pulsar broker To check whether the containers are successfully connected to the network, enter the `docker network inspect pulsar` command. -For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal). +For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal.md). diff --git a/site2/website/versioned_docs/version-2.7.3/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.7.3/deploy-kubernetes.md index 248d840c3016d..12b92350e03e8 100644 --- a/site2/website/versioned_docs/version-2.7.3/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.7.3/deploy-kubernetes.md @@ -6,6 +6,6 @@ original_id: deploy-kubernetes --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). diff --git a/site2/website/versioned_docs/version-2.7.3/deploy-monitoring.md b/site2/website/versioned_docs/version-2.7.3/deploy-monitoring.md index ab3735a0829b5..efc4e3c8ad9df 100644 --- a/site2/website/versioned_docs/version-2.7.3/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.7.3/deploy-monitoring.md @@ -86,7 +86,7 @@ Those metrics are added in the Prometheus interface, you can monitor and check t You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy). +When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy.md). ## Dashboards @@ -94,7 +94,7 @@ When you collect time series statistics, the major problem is to make sure the n ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager). +The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.7.3/develop-schema.md b/site2/website/versioned_docs/version-2.7.3/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.7.3/develop-schema.md +++ b/site2/website/versioned_docs/version-2.7.3/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.7.3/functions-deploy.md b/site2/website/versioned_docs/version-2.7.3/functions-deploy.md index 51f1140693dbe..d9496b3ed5b48 100644 --- a/site2/website/versioned_docs/version-2.7.3/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.7.3/functions-deploy.md @@ -9,12 +9,12 @@ original_id: functions-deploy To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface diff --git a/site2/website/versioned_docs/version-2.7.3/functions-develop.md b/site2/website/versioned_docs/version-2.7.3/functions-develop.md index 8c7e828153979..0911074c5381f 100644 --- a/site2/website/versioned_docs/version-2.7.3/functions-develop.md +++ b/site2/website/versioned_docs/version-2.7.3/functions-develop.md @@ -1163,7 +1163,7 @@ public class MetricRecorderFunction implements Function { ``` -> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring) guide. +> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring.md) guide.     @@ -1194,11 +1194,11 @@ Currently, the feature is not available in Go. ```` ### Access metrics -To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring) in Pulsar. +To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring.md) in Pulsar. ## Security -If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker). For more details, refer to [Security settings](functions-worker.md#security-settings). +If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings). Pulsar Functions can support the following providers: diff --git a/site2/website/versioned_docs/version-2.7.3/functions-overview.md b/site2/website/versioned_docs/version-2.7.3/functions-overview.md index 25dc6029acf70..816d301e0fd0e 100644 --- a/site2/website/versioned_docs/version-2.7.3/functions-overview.md +++ b/site2/website/versioned_docs/version-2.7.3/functions-overview.md @@ -164,7 +164,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.7.3/functions-package.md b/site2/website/versioned_docs/version-2.7.3/functions-package.md index d7a5b3ec20b60..ac25cb255e110 100644 --- a/site2/website/versioned_docs/version-2.7.3/functions-package.md +++ b/site2/website/versioned_docs/version-2.7.3/functions-package.md @@ -15,7 +15,7 @@ Currently, the window function is not available in Python and Go. ## Prerequisite -Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm). +Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm.md). To check whether the Docker image starts, you can use the `docker ps` command. diff --git a/site2/website/versioned_docs/version-2.7.3/functions-worker.md b/site2/website/versioned_docs/version-2.7.3/functions-worker.md index fa1467aa5fd30..9b603e88de609 100644 --- a/site2/website/versioned_docs/version-2.7.3/functions-worker.md +++ b/site2/website/versioned_docs/version-2.7.3/functions-worker.md @@ -4,7 +4,7 @@ title: Deploy and manage functions worker sidebar_label: "Setup: Pulsar Functions Worker" original_id: functions-worker --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -180,7 +180,7 @@ brokerClientTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -200,7 +200,7 @@ authenticationProviders: [ provider1, provider2 ] ``` For *TLS Authentication* provider, follow the example below to add the necessary settings. -See [TLS Authentication](security-tls-authentication) for more details. +See [TLS Authentication](security-tls-authentication.md) for more details. ``` @@ -224,7 +224,7 @@ properties: ``` For *Token Authentication* provider, add necessary settings for `properties` if needed. -See [Token Authentication](security-jwt) for more details. +See [Token Authentication](security-jwt.md) for more details. ``` @@ -262,7 +262,7 @@ superUserRoles: You can use the public and private key pair that the application configures to perform encryption. Only the consumers with a valid key can decrypt the encrypted messages. -To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption). +To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption.md). We include the relevant configuration information of `CryptoConfig` into `ProducerConfig`. The specific configurable field information about `CryptoConfig` is as follows: diff --git a/site2/website/versioned_docs/version-2.7.3/getting-started-docker.md b/site2/website/versioned_docs/version-2.7.3/getting-started-docker.md index b9f12f0b52688..4f20971d75330 100644 --- a/site2/website/versioned_docs/version-2.7.3/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.7.3/getting-started-docker.md @@ -50,8 +50,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -110,7 +110,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.7.3/getting-started-helm.md b/site2/website/versioned_docs/version-2.7.3/getting-started-helm.md index 65a9fc04fe568..440087c275c05 100644 --- a/site2/website/versioned_docs/version-2.7.3/getting-started-helm.md +++ b/site2/website/versioned_docs/version-2.7.3/getting-started-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.7.3/getting-started-pulsar.md b/site2/website/versioned_docs/version-2.7.3/getting-started-pulsar.md index 11c5e66cb62c2..752590f57b558 100644 --- a/site2/website/versioned_docs/version-2.7.3/getting-started-pulsar.md +++ b/site2/website/versioned_docs/version-2.7.3/getting-started-pulsar.md @@ -5,13 +5,13 @@ sidebar_label: "Pulsar 2.0" original_id: pulsar-2.0 --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.7.3/getting-started-standalone.md b/site2/website/versioned_docs/version-2.7.3/getting-started-standalone.md index 4dfcf27b54e62..e94bb4bb6b9c3 100644 --- a/site2/website/versioned_docs/version-2.7.3/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.7.3/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -59,7 +59,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.7.3/helm-install.md b/site2/website/versioned_docs/version-2.7.3/helm-install.md index edb12260f010e..1f4d5eb69d5dd 100644 --- a/site2/website/versioned_docs/version-2.7.3/helm-install.md +++ b/site2/website/versioned_docs/version-2.7.3/helm-install.md @@ -21,7 +21,7 @@ Before deploying Pulsar, you need to prepare your environment. ### Tools -Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools) on your computer. +Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools.md) on your computer. ## Cloud cluster preparation @@ -37,8 +37,8 @@ To create and connect to the Kubernetes cluster, follow the instructions: ## Pulsar deployment -Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy). +Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy.md). ## Pulsar upgrade -To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade). +To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade.md). diff --git a/site2/website/versioned_docs/version-2.7.3/helm-overview.md b/site2/website/versioned_docs/version-2.7.3/helm-overview.md index 9cde85e2d18d5..385d535e319b6 100644 --- a/site2/website/versioned_docs/version-2.7.3/helm-overview.md +++ b/site2/website/versioned_docs/version-2.7.3/helm-overview.md @@ -52,9 +52,9 @@ It includes support for: ## Pulsar Helm chart quick start -To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install). +This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install.md). ## Troubleshooting @@ -64,7 +64,7 @@ We have done our best to make these charts as seamless as possible. Occasionally The Apache Pulsar Helm chart contains all required dependencies. -If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm) for your first iteration. +If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm.md) for your first iteration. 1. [Preparation](helm-prepare.md) 2. [Deployment](helm-deploy.md) @@ -82,7 +82,7 @@ helm upgrade apache/pulsar -f pulsar.yaml ``` -For more detailed information, see [Upgrading](helm-upgrade). +For more detailed information, see [Upgrading](helm-upgrade.md). ## Uninstallation diff --git a/site2/website/versioned_docs/version-2.7.3/helm-prepare.md b/site2/website/versioned_docs/version-2.7.3/helm-prepare.md index 95d7e668f9eca..5e9f2f9ef4f68 100644 --- a/site2/website/versioned_docs/version-2.7.3/helm-prepare.md +++ b/site2/website/versioned_docs/version-2.7.3/helm-prepare.md @@ -89,4 +89,4 @@ PROJECT= USE_LOCAL_SSD=true LOCAL_SSD_COUNT= ## Next Steps -Continue with the [installation of the chart](helm-deploy) once you have the cluster up and running. +Continue with the [installation of the chart](helm-deploy.md) once you have the cluster up and running. diff --git a/site2/website/versioned_docs/version-2.7.3/helm-tools.md b/site2/website/versioned_docs/version-2.7.3/helm-tools.md index 6894608b03473..a7e4017b4fc4f 100644 --- a/site2/website/versioned_docs/version-2.7.3/helm-tools.md +++ b/site2/website/versioned_docs/version-2.7.3/helm-tools.md @@ -25,7 +25,7 @@ You can get Helm from the project's [releases page](https://github.com/helm/helm ### Next steps -Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare). +Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare.md). ## Additional information diff --git a/site2/website/versioned_docs/version-2.7.3/helm-upgrade.md b/site2/website/versioned_docs/version-2.7.3/helm-upgrade.md index e39cf97407355..7d671e6bfb3c1 100644 --- a/site2/website/versioned_docs/version-2.7.3/helm-upgrade.md +++ b/site2/website/versioned_docs/version-2.7.3/helm-upgrade.md @@ -20,7 +20,7 @@ You can retrieve your previous `--set` arguments cleanly, with `helm get values To upgrade Apache Pulsar to a newer version, follow these steps: 1. Check the change log for the specific version you would like to upgrade to. -2. Go through [deployment documentation](helm-deploy) step by step. +2. Go through [deployment documentation](helm-deploy.md) step by step. 3. Extract your previous `--set` arguments with the following command. ```bash diff --git a/site2/website/versioned_docs/version-2.7.3/io-cdc.md b/site2/website/versioned_docs/version-2.7.3/io-cdc.md index 20f16ae0d6904..e6e662884826d 100644 --- a/site2/website/versioned_docs/version-2.7.3/io-cdc.md +++ b/site2/website/versioned_docs/version-2.7.3/io-cdc.md @@ -13,8 +13,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.7.3/io-develop.md b/site2/website/versioned_docs/version-2.7.3/io-develop.md index 4a815b3b9d6f6..7a356662560c9 100644 --- a/site2/website/versioned_docs/version-2.7.3/io-develop.md +++ b/site2/website/versioned_docs/version-2.7.3/io-develop.md @@ -8,15 +8,15 @@ original_id: io-develop This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -163,7 +163,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -192,7 +192,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). diff --git a/site2/website/versioned_docs/version-2.7.3/io-overview.md b/site2/website/versioned_docs/version-2.7.3/io-overview.md index 68df2472d6334..b538604edc46f 100644 --- a/site2/website/versioned_docs/version-2.7.3/io-overview.md +++ b/site2/website/versioned_docs/version-2.7.3/io-overview.md @@ -158,7 +158,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.7.3/io-quickstart.md b/site2/website/versioned_docs/version-2.7.3/io-quickstart.md index b8fea1560bdeb..8474c93f51336 100644 --- a/site2/website/versioned_docs/version-2.7.3/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.7.3/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you are able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you are able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However, all the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -109,7 +109,7 @@ This section demonstrates how to connect Pulsar to Cassandra. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink.md). ::: @@ -236,11 +236,11 @@ You can create a configuration file through one of the following methods. ``` -For more information, see [Cassandra sink connector](io-cassandra-sink). +For more information, see [Cassandra sink connector](io-cassandra-sink.md). ### Create a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on them. Run the following command to create a Cassandra sink connector with sink type _cassandra_ and the config file _examples/cassandra-sink.yml_ created previously. @@ -267,7 +267,7 @@ as a Pulsar Function and writes the messages produced in the topic _test_cassand ### Inspect a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * Get the information of a Cassandra sink. @@ -431,7 +431,7 @@ to monitor a connector and perform other operations on it. ### Delete a Cassandra Sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. ```bash @@ -454,7 +454,7 @@ This section demonstrates how to connect Pulsar to PostgreSQL. ::: ->For more information, see [JDBC sink connector](io-jdbc-sink). +>For more information, see [JDBC sink connector](io-jdbc-sink.md). ### Setup a PostgreSQL cluster @@ -614,7 +614,7 @@ In this section, you need to configure a JDBC sink connector. ### Create a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on it. This example creates a sink connector and specifies the desired information. @@ -660,7 +660,7 @@ The sink has been created successfully if the following message appears. ### Inspect a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * List all running JDBC sink(s). @@ -780,7 +780,7 @@ to monitor a connector and perform other operations on it. ### Stop a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to stop a connector and perform other operations on it. ```bash @@ -808,7 +808,7 @@ The sink instance has been stopped successfully if the following message disappe ### Restart a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to restart a connector and perform other operations on it. ```bash @@ -844,7 +844,7 @@ Note that `pulsar-admin sinks localrun options` **runs a sink connector locally* ### Update a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to update a connector and perform other operations on it. This example updates the parallelism of the _pulsar-postgres-jdbc-sink_ sink connector to 2. @@ -912,7 +912,7 @@ The result shows that the parallelism is 2. ### Delete a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. This example deletes the _pulsar-postgres-jdbc-sink_ sink connector. diff --git a/site2/website/versioned_docs/version-2.7.3/io-use.md b/site2/website/versioned_docs/version-2.7.3/io-use.md index b6409d4420ad7..da9ed746c4d37 100644 --- a/site2/website/versioned_docs/version-2.7.3/io-use.md +++ b/site2/website/versioned_docs/version-2.7.3/io-use.md @@ -15,7 +15,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.7.3/kubernetes-helm.md b/site2/website/versioned_docs/version-2.7.3/kubernetes-helm.md index a9f768e6a96b3..ea92a0968cd7d 100644 --- a/site2/website/versioned_docs/version-2.7.3/kubernetes-helm.md +++ b/site2/website/versioned_docs/version-2.7.3/kubernetes-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.7.3/reference-cli-tools.md b/site2/website/versioned_docs/version-2.7.3/reference-cli-tools.md index d17bd3efd9790..2bc3cc56130f5 100644 --- a/site2/website/versioned_docs/version-2.7.3/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.7.3/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.7.3/schema-get-started.md b/site2/website/versioned_docs/version-2.7.3/schema-get-started.md index 97e270586a26d..afacb0fa51f2e 100644 --- a/site2/website/versioned_docs/version-2.7.3/schema-get-started.md +++ b/site2/website/versioned_docs/version-2.7.3/schema-get-started.md @@ -17,7 +17,7 @@ Applications typically adopt one of the following approaches to guarantee type s #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ### Client-side approach diff --git a/site2/website/versioned_docs/version-2.7.3/schema-manage.md b/site2/website/versioned_docs/version-2.7.3/schema-manage.md index 98105408fb37d..5984fdbeaaa5f 100644 --- a/site2/website/versioned_docs/version-2.7.3/schema-manage.md +++ b/site2/website/versioned_docs/version-2.7.3/schema-manage.md @@ -160,7 +160,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.7.3/schema-understand.md b/site2/website/versioned_docs/version-2.7.3/schema-understand.md index 333c902b58319..d17c6168da59f 100644 --- a/site2/website/versioned_docs/version-2.7.3/schema-understand.md +++ b/site2/website/versioned_docs/version-2.7.3/schema-understand.md @@ -380,7 +380,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```java diff --git a/site2/website/versioned_docs/version-2.7.3/security-athenz.md b/site2/website/versioned_docs/version-2.7.3/security-athenz.md index 947c3f470be46..8a39fe25316d0 100644 --- a/site2/website/versioned_docs/version-2.7.3/security-athenz.md +++ b/site2/website/versioned_docs/version-2.7.3/security-athenz.md @@ -76,7 +76,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.7.3/security-authorization.md b/site2/website/versioned_docs/version-2.7.3/security-authorization.md index e6785873f0e10..7ac09b6f439ea 100644 --- a/site2/website/versioned_docs/version-2.7.3/security-authorization.md +++ b/site2/website/versioned_docs/version-2.7.3/security-authorization.md @@ -27,7 +27,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -59,7 +59,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -87,7 +87,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication diff --git a/site2/website/versioned_docs/version-2.7.3/security-bouncy-castle.md b/site2/website/versioned_docs/version-2.7.3/security-bouncy-castle.md index abc25c71b8ae2..be937055d8e31 100644 --- a/site2/website/versioned_docs/version-2.7.3/security-bouncy-castle.md +++ b/site2/website/versioned_docs/version-2.7.3/security-bouncy-castle.md @@ -16,7 +16,7 @@ In Pulsar, security and crypto have dependencies on BouncyCastle Jars. For the d `Bouncy Castle` provides both [FIPS](https://www.bouncycastle.org/fips_faq.html) and non-FIPS version. But in a JVM, you can not include both of the 2 versions, and you need to exclude the current version before include the other. -In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. +In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption.md). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. ## How BouncyCastle modules packaged in Pulsar diff --git a/site2/website/versioned_docs/version-2.7.3/security-extending.md b/site2/website/versioned_docs/version-2.7.3/security-extending.md index c088e3a2c1c2c..e7484453b8beb 100644 --- a/site2/website/versioned_docs/version-2.7.3/security-extending.md +++ b/site2/website/versioned_docs/version-2.7.3/security-extending.md @@ -9,7 +9,7 @@ Pulsar provides a way to use custom authentication and authorization mechanisms. ## Authentication -Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview). +Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview.md). You can use a custom authentication mechanism by providing the implementation in the form of two plugins. One plugin is for the Client library and the other plugin is for the Pulsar Proxy and/or Pulsar Broker to validate the credentials. diff --git a/site2/website/versioned_docs/version-2.7.3/security-jwt.md b/site2/website/versioned_docs/version-2.7.3/security-jwt.md index 656ab362f71b0..91117bedff753 100644 --- a/site2/website/versioned_docs/version-2.7.3/security-jwt.md +++ b/site2/website/versioned_docs/version-2.7.3/security-jwt.md @@ -32,11 +32,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -303,7 +303,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.7.3/security-kerberos.md b/site2/website/versioned_docs/version-2.7.3/security-kerberos.md index 670586a9d1691..c49fa3bea1fce 100644 --- a/site2/website/versioned_docs/version-2.7.3/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.7.3/security-kerberos.md @@ -373,7 +373,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.7.3/security-overview.md b/site2/website/versioned_docs/version-2.7.3/security-overview.md index c7823104d145b..a1664fcb8b450 100644 --- a/site2/website/versioned_docs/version-2.7.3/security-overview.md +++ b/site2/website/versioned_docs/version-2.7.3/security-overview.md @@ -21,7 +21,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.7.3/security-tls-authentication.md b/site2/website/versioned_docs/version-2.7.3/security-tls-authentication.md index 79aa8ea9fd13d..61c8aa43e1421 100644 --- a/site2/website/versioned_docs/version-2.7.3/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.7.3/security-tls-authentication.md @@ -7,9 +7,9 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -101,7 +101,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -121,7 +121,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.7.3/security-tls-keystore.md b/site2/website/versioned_docs/version-2.7.3/security-tls-keystore.md index 062d903e0e2ca..cb14afd1e4080 100644 --- a/site2/website/versioned_docs/version-2.7.3/security-tls-keystore.md +++ b/site2/website/versioned_docs/version-2.7.3/security-tls-keystore.md @@ -7,7 +7,7 @@ original_id: security-tls-keystore ## Overview -Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication) between clients and Apache Pulsar service. +Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication.md) between clients and Apache Pulsar service. By default it uses PEM format file configuration. This page tries to describe use [KeyStore](https://en.wikipedia.org/wiki/Java_KeyStore) type configure for TLS. @@ -165,7 +165,7 @@ This is similar to [TLS encryption configuing for client with PEM type](security For a a minimal configuration, user need to provide the TrustStore information. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties @@ -258,7 +258,7 @@ webSocketServiceEnabled=false Besides the TLS encryption configuring. The main work is configuring the KeyStore, which contains a valid CN as client role, for client. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties diff --git a/site2/website/versioned_docs/version-2.7.3/security-tls-transport.md b/site2/website/versioned_docs/version-2.7.3/security-tls-transport.md index 12c4f2e22812b..31dd72589067e 100644 --- a/site2/website/versioned_docs/version-2.7.3/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.7.3/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,9 +27,9 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). -`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ## Create TLS certificates @@ -130,7 +130,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Broker Configuration -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -201,7 +201,7 @@ The examples below show that hostname verification is disabled for the CLI tools ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.7.3/security-token-admin.md b/site2/website/versioned_docs/version-2.7.3/security-token-admin.md index 2173a6cb1d11d..a5d69da9f0602 100644 --- a/site2/website/versioned_docs/version-2.7.3/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.7.3/security-token-admin.md @@ -163,7 +163,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.7.3/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.7.3/sql-deployment-configurations.md index c6d7c88a07e94..cc8bdbcd4fd51 100644 --- a/site2/website/versioned_docs/version-2.7.3/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.7.3/sql-deployment-configurations.md @@ -64,7 +64,7 @@ Since Pulsar SQL is powered by [Presto](https://prestosql.io), the configuration :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.7.3/sql-overview.md b/site2/website/versioned_docs/version-2.7.3/sql-overview.md index 3303923c6d237..e4dee1284869b 100644 --- a/site2/website/versioned_docs/version-2.7.3/sql-overview.md +++ b/site2/website/versioned_docs/version-2.7.3/sql-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Overview" original_id: sql-overview --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.7.3/standalone-docker.md b/site2/website/versioned_docs/version-2.7.3/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.7.3/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.7.3/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.7.3/standalone.md b/site2/website/versioned_docs/version-2.7.3/standalone.md index d8dd985f30d84..fdec1cc3d2bf3 100644 --- a/site2/website/versioned_docs/version-2.7.3/standalone.md +++ b/site2/website/versioned_docs/version-2.7.3/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -59,7 +59,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.7.3/tiered-storage-overview.md b/site2/website/versioned_docs/version-2.7.3/tiered-storage-overview.md index 7c6528ce2b36c..c635034f463b4 100644 --- a/site2/website/versioned_docs/version-2.7.3/tiered-storage-overview.md +++ b/site2/website/versioned_docs/version-2.7.3/tiered-storage-overview.md @@ -13,9 +13,9 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws). + - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws.md). - - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs). + - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs.md). ::: @@ -25,7 +25,7 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem). + For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem.md). ::: diff --git a/site2/website/versioned_docs/version-2.7.4/admin-api-brokers.md b/site2/website/versioned_docs/version-2.7.4/admin-api-brokers.md index 751bc14983205..61047c4d631d1 100644 --- a/site2/website/versioned_docs/version-2.7.4/admin-api-brokers.md +++ b/site2/website/versioned_docs/version-2.7.4/admin-api-brokers.md @@ -18,7 +18,7 @@ Pulsar brokers consist of two components: [Brokers](reference-terminology.md#broker) can be managed via: -* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/brokers` endpoint of the admin {@inject: rest:REST:/} API * The `brokers` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin.html} object in the [Java API](client-libraries-java.md) @@ -123,7 +123,7 @@ One way to configure a Pulsar [broker](reference-terminology.md#broker) is to su But since all broker configuration in Pulsar is stored in ZooKeeper, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values. -* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. +* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. * In the Pulsar admin {@inject: rest:REST:/} API, dynamic configuration is managed through the `/admin/v2/brokers/configuration` endpoint. ### Update dynamic configuration diff --git a/site2/website/versioned_docs/version-2.7.4/admin-api-clusters.md b/site2/website/versioned_docs/version-2.7.4/admin-api-clusters.md index f3af175295514..79508763e1db0 100644 --- a/site2/website/versioned_docs/version-2.7.4/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.7.4/admin-api-clusters.md @@ -16,7 +16,7 @@ servers (aka [bookies](reference-terminology.md#bookie)), and a [ZooKeeper](http Clusters can be managed via: -* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/clusters` endpoint of the admin {@inject: rest:REST:/} API * The `clusters` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md) @@ -79,7 +79,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -103,11 +103,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block -If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: +If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin.md) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: |Name|Description|Default| |----|-----------|-------| diff --git a/site2/website/versioned_docs/version-2.7.4/admin-api-partitioned-topics.md b/site2/website/versioned_docs/version-2.7.4/admin-api-partitioned-topics.md index 28338e8cc4a69..4582ba14772d8 100644 --- a/site2/website/versioned_docs/version-2.7.4/admin-api-partitioned-topics.md +++ b/site2/website/versioned_docs/version-2.7.4/admin-api-partitioned-topics.md @@ -5,4 +5,4 @@ sidebar_label: "Partitioned topics" original_id: admin-api-partitioned-topics --- -For details of the content, refer to [manage topics](admin-api-topics). +For details of the content, refer to [manage topics](admin-api-topics.md). diff --git a/site2/website/versioned_docs/version-2.7.4/admin-api-persistent-topics.md b/site2/website/versioned_docs/version-2.7.4/admin-api-persistent-topics.md index 596e0358227ea..13e24f1d9eba4 100644 --- a/site2/website/versioned_docs/version-2.7.4/admin-api-persistent-topics.md +++ b/site2/website/versioned_docs/version-2.7.4/admin-api-persistent-topics.md @@ -5,4 +5,4 @@ sidebar_label: "Persistent topics" original_id: admin-api-persistent-topics --- -For details of the content, refer to [manage topics](admin-api-topics). +For details of the content, refer to [manage topics](admin-api-topics.md). diff --git a/site2/website/versioned_docs/version-2.7.4/admin-api-tenants.md b/site2/website/versioned_docs/version-2.7.4/admin-api-tenants.md index 63afe58620c62..d78aa2e55f4c3 100644 --- a/site2/website/versioned_docs/version-2.7.4/admin-api-tenants.md +++ b/site2/website/versioned_docs/version-2.7.4/admin-api-tenants.md @@ -11,7 +11,7 @@ import TabItem from '@theme/TabItem'; ```` -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -108,7 +108,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.7.4/client-libraries-cgo.md b/site2/website/versioned_docs/version-2.7.4/client-libraries-cgo.md index c79f7bb965473..f352f942b7714 100644 --- a/site2/website/versioned_docs/version-2.7.4/client-libraries-cgo.md +++ b/site2/website/versioned_docs/version-2.7.4/client-libraries-cgo.md @@ -24,7 +24,7 @@ Currently, the following Go clients are maintained in two repositories. ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Install go package @@ -57,7 +57,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -75,7 +75,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -236,14 +236,14 @@ Parameter | Description | Default `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | `Properties`| Attach a set of application defined properties to the producer. This properties will be visible in the topic stats | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | Set the number of max pending messages across all the partitions. This setting will be used to lower the max pending messages for each partition `MaxPendingMessages(int)`, if the total exceeds the configured value.| `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | `Batching` | Control whether automatic batching of messages is enabled for the producer. | false `BatchingMaxPublishDelay` | Set the time period within which the messages sent will be batched (default: 1ms) if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or until | 1ms `BatchingMaxMessages` | Set the maximum number of messages permitted in a batch. (default: 1000) If set to a value greater than 1, messages will be queued until this threshold is reached or batch interval has elapsed | 1000 @@ -512,7 +512,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.7.4/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.7.4/client-libraries-cpp.md index bb52d9effa404..684df5fd2d825 100644 --- a/site2/website/versioned_docs/version-2.7.4/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.7.4/client-libraries-cpp.md @@ -261,7 +261,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema -This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started). +This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started.md). ### Create producer with Avro schema diff --git a/site2/website/versioned_docs/version-2.7.4/client-libraries-dotnet.md b/site2/website/versioned_docs/version-2.7.4/client-libraries-dotnet.md index ade664cd95277..b574fa0b2e5ed 100644 --- a/site2/website/versioned_docs/version-2.7.4/client-libraries-dotnet.md +++ b/site2/website/versioned_docs/version-2.7.4/client-libraries-dotnet.md @@ -170,7 +170,7 @@ var client = PulsarClient.Builder() Currently, the Pulsar C# client supports the TLS (Transport Layer Security) and JWT (JSON Web Token) authentication. -If you have followed [Authentication using TLS](security-tls-authentication), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: +If you have followed [Authentication using TLS](security-tls-authentication.md), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: 1. Create an unencrypted and password-less pfx file. diff --git a/site2/website/versioned_docs/version-2.7.4/client-libraries-go.md b/site2/website/versioned_docs/version-2.7.4/client-libraries-go.md index 97705a643c049..6c05e1d733422 100644 --- a/site2/website/versioned_docs/version-2.7.4/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.7.4/client-libraries-go.md @@ -35,7 +35,7 @@ import "github.com/apache/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -53,7 +53,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -680,7 +680,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker @@ -700,7 +700,7 @@ opts := pulsar.ClientOptions{ ## OAuth2 authentication -To use [OAuth2 authentication](security-oauth2), you'll need to configure your client to perform the following operations. +To use [OAuth2 authentication](security-oauth2.md), you'll need to configure your client to perform the following operations. This example shows how to configure OAuth2 authentication. ```go diff --git a/site2/website/versioned_docs/version-2.7.4/client-libraries-java.md b/site2/website/versioned_docs/version-2.7.4/client-libraries-java.md index bbe9473622fbe..ed7ce010e7550 100644 --- a/site2/website/versioned_docs/version-2.7.4/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.7.4/client-libraries-java.md @@ -5,7 +5,7 @@ sidebar_label: "Java" original_id: client-libraries-java --- -You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), and [reader](#reader) of a Java client are thread-safe. @@ -14,10 +14,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Includes both `pulsar-client` and `pulsar-client-admin`

    Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.

    In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -57,7 +57,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -83,7 +83,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -114,7 +114,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -237,7 +237,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -822,7 +822,7 @@ Total hash range size is 65536, so the max end of the range should be less than ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -936,7 +936,7 @@ The following schema formats are currently available for Java: ## Authentication -Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2). You can use the Pulsar Java client with all of them. +Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2.md). You can use the Pulsar Java client with all of them. ### TLS Authentication @@ -963,7 +963,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` @@ -1000,7 +1000,7 @@ PulsarClient client = PulsarClient.builder() ### Oauth2 -The following example shows how to use [Oauth2](security-oauth2) as an authentication provider for the Pulsar Java client. +The following example shows how to use [Oauth2](security-oauth2.md) as an authentication provider for the Pulsar Java client. You can use the factory method to configure authentication for Pulsar Java client. diff --git a/site2/website/versioned_docs/version-2.7.4/client-libraries-node.md b/site2/website/versioned_docs/version-2.7.4/client-libraries-node.md index 12c1bdf590f1a..db851088a571f 100644 --- a/site2/website/versioned_docs/version-2.7.4/client-libraries-node.md +++ b/site2/website/versioned_docs/version-2.7.4/client-libraries-node.md @@ -46,7 +46,7 @@ Also, this library works only in Node.js 10.x or later because it uses the [`nod ::: ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here is an example for `localhost`: @@ -64,7 +64,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication), the URL will look like something like this: +If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication.md), the URL will look like something like this: ```http @@ -99,7 +99,7 @@ The following configurable parameters are available for Pulsar clients: | Parameter | Description | Default | | :-------- | :---------- | :------ | | `serviceUrl` | The connection URL for the Pulsar cluster. See [above](#connection-urls) for more info. | | -| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication) for more info. | | +| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication.md) for more info. | | | `operationTimeoutSeconds` | The timeout for Node.js client operations (creating producers, subscribing to and unsubscribing from [topics](reference-terminology.md#topic)). Retries will occur until this threshold is reached, at which point the operation will fail. | 30 | | `ioThreads` | The number of threads to use for handling connections to Pulsar [brokers](reference-terminology.md#broker). | 1 | | `messageListenerThreads` | The number of threads used by message listeners ([consumers](#consumers) and [readers](#readers)). | 1 | @@ -150,7 +150,7 @@ Pulsar Node.js producers have the following methods available: | :-------- | :---------- | :------ | | `topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer publishes messages. The topic format is `` or `//`. For example, `sample/ns1/my-topic`. | | | `producerName` | A name for the producer. If you do not explicitly assign a name, Pulsar will automatically generate a globally unique name. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | | -| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30000 | +| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30000 | | `initialSequenceId` | The initial sequence ID of the message. When producer send message, add sequence ID to message. The ID is increased each time to send. | | | `maxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `send` method will fail *unless* `blockIfQueueFull` is set to `true`. | 1000 | | `maxPendingMessagesAcrossPartitions` | The maximum size of the sum of partition's pending queue. | 50000 | diff --git a/site2/website/versioned_docs/version-2.7.4/client-libraries-python.md b/site2/website/versioned_docs/version-2.7.4/client-libraries-python.md index 7d44db909ca9f..e7ea7f01813f8 100644 --- a/site2/website/versioned_docs/version-2.7.4/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.7.4/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. diff --git a/site2/website/versioned_docs/version-2.7.4/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.7.4/client-libraries-websocket.md index cdf87d70aa905..4906a93ec0e6f 100644 --- a/site2/website/versioned_docs/version-2.7.4/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.7.4/client-libraries-websocket.md @@ -5,14 +5,14 @@ sidebar_label: "WebSocket" original_id: client-libraries-websocket --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.7.4/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.7.4/concepts-architecture-overview.md index c65f9149ceda5..ad3f04858b25a 100644 --- a/site2/website/versioned_docs/version-2.7.4/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.7.4/concepts-architecture-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Architecture" original_id: concepts-architecture-overview --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -17,20 +17,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -40,9 +40,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -130,17 +130,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -148,7 +148,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.7.4/concepts-authentication.md b/site2/website/versioned_docs/version-2.7.4/concepts-authentication.md index b375ecb16a2d8..f6307890c904a 100644 --- a/site2/website/versioned_docs/version-2.7.4/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.7.4/concepts-authentication.md @@ -5,5 +5,5 @@ sidebar_label: "Authentication and Authorization" original_id: concepts-authentication --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.7.4/concepts-clients.md b/site2/website/versioned_docs/version-2.7.4/concepts-clients.md index b68f76a2d8b08..4040624f7d636 100644 --- a/site2/website/versioned_docs/version-2.7.4/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.7.4/concepts-clients.md @@ -5,12 +5,12 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -38,7 +38,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.7.4/concepts-messaging.md b/site2/website/versioned_docs/version-2.7.4/concepts-messaging.md index a4ea0637c94eb..4b03824755273 100644 --- a/site2/website/versioned_docs/version-2.7.4/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.7.4/concepts-messaging.md @@ -21,8 +21,8 @@ Messages are the basic "unit" of Pulsar. The following table lists the component Component | Description :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer who produces the message. If you do not specify a producer name, the default name is used. Sequence ID | Each Pulsar message belongs to an ordered sequence on its topic. The sequence ID of the message is its order in that sequence. @@ -30,7 +30,7 @@ Publish time | The timestamp of when the message is published. The timestamp is Event time | An optional timestamp attached to a message by applications. For example, applications attach a timestamp on when the message is processed. If nothing is set to event time, the value is `0`. TypedMessageBuilder | It is used to construct a message. You can set message properties such as the message key, message value with `TypedMessageBuilder`.
    When you set `TypedMessageBuilder`, set the key as a string. If you set the key as other types, for example, an AVRO object, the key is sent as bytes, and it is difficult to get the AVRO object back on the consumer. -> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol). +> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -107,7 +107,7 @@ Messages are received from [brokers](reference-terminology.md#broker) either syn ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -344,7 +344,7 @@ A [subscription](#subscriptions) can have one or more consumers. When a consumer #### When to use -By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry). +By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry.md). #### How to use @@ -448,7 +448,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -460,7 +460,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -498,7 +498,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -553,7 +553,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -576,12 +576,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication.md). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.7.4/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.7.4/concepts-multi-tenancy.md index be752ccbbf9ca..c4acb21acb743 100644 --- a/site2/website/versioned_docs/version-2.7.4/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.7.4/concepts-multi-tenancy.md @@ -5,7 +5,7 @@ sidebar_label: "Multi Tenancy" original_id: concepts-multi-tenancy --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -21,7 +21,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -29,7 +29,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.7.4/concepts-overview.md b/site2/website/versioned_docs/version-2.7.4/concepts-overview.md index c1262b38daf16..e8a2f4b9d321a 100644 --- a/site2/website/versioned_docs/version-2.7.4/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.7.4/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.7.4/concepts-replication.md b/site2/website/versioned_docs/version-2.7.4/concepts-replication.md index 6e23962e99771..799f0eb4d92c6 100644 --- a/site2/website/versioned_docs/version-2.7.4/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.7.4/concepts-replication.md @@ -5,5 +5,5 @@ sidebar_label: "Geo Replication" original_id: concepts-replication --- -Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that. +Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo.md) in Pulsar enables you to do that. diff --git a/site2/website/versioned_docs/version-2.7.4/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.7.4/concepts-tiered-storage.md index 0b45b0a2ed501..f6988e53a8cd4 100644 --- a/site2/website/versioned_docs/version-2.7.4/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.7.4/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.7.4/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.7.4/concepts-topic-compaction.md index e402c965384fd..34b7ed7fbbd31 100644 --- a/site2/website/versioned_docs/version-2.7.4/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.7.4/concepts-topic-compaction.md @@ -7,7 +7,7 @@ original_id: concepts-topic-compaction Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -24,7 +24,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.7.4/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.7.4/cookbooks-compaction.md index 6fcbc197f9ecb..dfa314727241a 100644 --- a/site2/website/versioned_docs/version-2.7.4/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.7.4/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.7.4/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.7.4/cookbooks-non-persistent.md index 391569a8cbc6b..178301e86eb8d 100644 --- a/site2/website/versioned_docs/version-2.7.4/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.7.4/cookbooks-non-persistent.md @@ -41,7 +41,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -55,7 +55,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.7.4/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.7.4/cookbooks-partitioned.md index 39de6de952689..f279f2265f666 100644 --- a/site2/website/versioned_docs/version-2.7.4/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.7.4/cookbooks-partitioned.md @@ -4,4 +4,4 @@ title: Partitioned topics sidebar_label: "Partitioned Topics" original_id: cookbooks-partitioned --- -For details of the content, refer to [manage topics](admin-api-topics). +For details of the content, refer to [manage topics](admin-api-topics.md). diff --git a/site2/website/versioned_docs/version-2.7.4/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.7.4/cookbooks-retention-expiry.md index 5295b4c5de4ae..23175d607b37e 100644 --- a/site2/website/versioned_docs/version-2.7.4/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.7.4/cookbooks-retention-expiry.md @@ -22,7 +22,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.7.4/deploy-aws.md b/site2/website/versioned_docs/version-2.7.4/deploy-aws.md index 1c8f1a08d7249..662e99ceb9b09 100644 --- a/site2/website/versioned_docs/version-2.7.4/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.7.4/deploy-aws.md @@ -5,7 +5,7 @@ sidebar_label: "Amazon Web Services" original_id: deploy-aws --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -210,7 +210,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.7.4/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.7.4/deploy-bare-metal-multi-cluster.md index 4ee6da37f0a90..74303a3ff7924 100644 --- a/site2/website/versioned_docs/version-2.7.4/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.7.4/deploy-bare-metal-multi-cluster.md @@ -9,17 +9,17 @@ original_id: deploy-bare-metal-multi-cluster 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using it in a startup or on a single team, you had better opt for a single cluster. For instructions on deploying a single cluster, -see the guide [here](deploy-bare-metal). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo). Deploying a multi-cluster Pulsar instance involves the following basic steps: +A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md). Deploying a multi-cluster Pulsar instance involves the following basic steps: * Deploying two separate [ZooKeeper](#deploy-zookeeper) quorums: a [local](#deploy-local-zookeeper) quorum for each cluster in the instance and a [configuration store](#configuration-store) quorum for instance-wide tasks * Initializing [cluster metadata](#cluster-metadata-initialization) for each cluster @@ -29,7 +29,7 @@ A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You If you want to deploy a single Pulsar cluster, see [Clusters and Brokers](getting-started-standalone.md#start-the-cluster). > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). ## System requirement Pulsar is currently available for **MacOS** and **Linux**. In order to use Pulsar, you need to install Java 8 from [Oracle download center](http://www.oracle.com/). @@ -67,7 +67,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -245,7 +245,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -364,17 +364,17 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). -You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > #### Service discovery already provided by many scheduling systems -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ### Service discovery setup -The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol) of Pulsar. +The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol.md) of Pulsar. To get started setting up the built-in service of discovery of Pulsar, you need to change a few parameters in the [`conf/discovery.conf`](reference-configuration.md#service-discovery) configuration file. Set the [`zookeeperServers`](reference-configuration.md#service-discovery-zookeeperServers) parameter to the ZooKeeper quorum connection string of the cluster and the [`configurationStoreServers`](reference-configuration.md#service-discovery-configurationStoreServers) setting to the [configuration store](reference-terminology.md#configuration-store) quorum connection string. @@ -399,7 +399,7 @@ $ bin/pulsar-daemon start discovery ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.7.4/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.7.4/deploy-bare-metal.md index 7b1850b3b21aa..1ea6286f6e533 100644 --- a/site2/website/versioned_docs/version-2.7.4/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.7.4/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using Pulsar in a startup or on a single team, it is simplest to opt for a single cluster. If you do need to run a multi-cluster Pulsar instance, -see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -99,7 +99,7 @@ The extracted directory contains the following subdirectories: Directory | Contains :---------|:-------- -`bin` |[command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` |[command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `data` | The data storage directory that ZooKeeper and BookKeeper use `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -261,9 +261,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (you had better not use a different port). -`--web-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). +`--web-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should not use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (you had better not use a different port). -`--broker-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). +`--broker-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). > If you do not have a DNS server, you can use multi-host format in the service URL with the following settings: @@ -403,7 +403,7 @@ webServicePortTls=8443 ### Enable Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -421,7 +421,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker). +If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker.md). ### Start Brokers diff --git a/site2/website/versioned_docs/version-2.7.4/deploy-dcos.md b/site2/website/versioned_docs/version-2.7.4/deploy-dcos.md index f5f8d1faa7dbe..952d5f47e30fa 100644 --- a/site2/website/versioned_docs/version-2.7.4/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.7.4/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -If you want to enable all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of +If you want to enable all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.7.4/deploy-docker.md b/site2/website/versioned_docs/version-2.7.4/deploy-docker.md index 64a0939aa787b..8348d78deb237 100644 --- a/site2/website/versioned_docs/version-2.7.4/deploy-docker.md +++ b/site2/website/versioned_docs/version-2.7.4/deploy-docker.md @@ -57,4 +57,4 @@ docker network connect pulsar broker To check whether the containers are successfully connected to the network, enter the `docker network inspect pulsar` command. -For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal). +For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal.md). diff --git a/site2/website/versioned_docs/version-2.7.4/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.7.4/deploy-kubernetes.md index 248d840c3016d..12b92350e03e8 100644 --- a/site2/website/versioned_docs/version-2.7.4/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.7.4/deploy-kubernetes.md @@ -6,6 +6,6 @@ original_id: deploy-kubernetes --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). diff --git a/site2/website/versioned_docs/version-2.7.4/deploy-monitoring.md b/site2/website/versioned_docs/version-2.7.4/deploy-monitoring.md index 4b4f0b40f4007..214502a7e8fea 100644 --- a/site2/website/versioned_docs/version-2.7.4/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.7.4/deploy-monitoring.md @@ -86,7 +86,7 @@ Those metrics are added in the Prometheus interface, you can monitor and check t You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy). +When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy.md). ## Dashboards @@ -94,7 +94,7 @@ When you collect time series statistics, the major problem is to make sure the n ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager). +The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.7.4/develop-schema.md b/site2/website/versioned_docs/version-2.7.4/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.7.4/develop-schema.md +++ b/site2/website/versioned_docs/version-2.7.4/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.7.4/functions-deploy.md b/site2/website/versioned_docs/version-2.7.4/functions-deploy.md index 51f1140693dbe..d9496b3ed5b48 100644 --- a/site2/website/versioned_docs/version-2.7.4/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.7.4/functions-deploy.md @@ -9,12 +9,12 @@ original_id: functions-deploy To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface diff --git a/site2/website/versioned_docs/version-2.7.4/functions-develop.md b/site2/website/versioned_docs/version-2.7.4/functions-develop.md index 8c7e828153979..0911074c5381f 100644 --- a/site2/website/versioned_docs/version-2.7.4/functions-develop.md +++ b/site2/website/versioned_docs/version-2.7.4/functions-develop.md @@ -1163,7 +1163,7 @@ public class MetricRecorderFunction implements Function { ``` -> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring) guide. +> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring.md) guide.     @@ -1194,11 +1194,11 @@ Currently, the feature is not available in Go. ```` ### Access metrics -To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring) in Pulsar. +To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring.md) in Pulsar. ## Security -If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker). For more details, refer to [Security settings](functions-worker.md#security-settings). +If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings). Pulsar Functions can support the following providers: diff --git a/site2/website/versioned_docs/version-2.7.4/functions-overview.md b/site2/website/versioned_docs/version-2.7.4/functions-overview.md index 25dc6029acf70..816d301e0fd0e 100644 --- a/site2/website/versioned_docs/version-2.7.4/functions-overview.md +++ b/site2/website/versioned_docs/version-2.7.4/functions-overview.md @@ -164,7 +164,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.7.4/functions-package.md b/site2/website/versioned_docs/version-2.7.4/functions-package.md index d7a5b3ec20b60..ac25cb255e110 100644 --- a/site2/website/versioned_docs/version-2.7.4/functions-package.md +++ b/site2/website/versioned_docs/version-2.7.4/functions-package.md @@ -15,7 +15,7 @@ Currently, the window function is not available in Python and Go. ## Prerequisite -Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm). +Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm.md). To check whether the Docker image starts, you can use the `docker ps` command. diff --git a/site2/website/versioned_docs/version-2.7.4/functions-worker.md b/site2/website/versioned_docs/version-2.7.4/functions-worker.md index fa1467aa5fd30..9b603e88de609 100644 --- a/site2/website/versioned_docs/version-2.7.4/functions-worker.md +++ b/site2/website/versioned_docs/version-2.7.4/functions-worker.md @@ -4,7 +4,7 @@ title: Deploy and manage functions worker sidebar_label: "Setup: Pulsar Functions Worker" original_id: functions-worker --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -180,7 +180,7 @@ brokerClientTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -200,7 +200,7 @@ authenticationProviders: [ provider1, provider2 ] ``` For *TLS Authentication* provider, follow the example below to add the necessary settings. -See [TLS Authentication](security-tls-authentication) for more details. +See [TLS Authentication](security-tls-authentication.md) for more details. ``` @@ -224,7 +224,7 @@ properties: ``` For *Token Authentication* provider, add necessary settings for `properties` if needed. -See [Token Authentication](security-jwt) for more details. +See [Token Authentication](security-jwt.md) for more details. ``` @@ -262,7 +262,7 @@ superUserRoles: You can use the public and private key pair that the application configures to perform encryption. Only the consumers with a valid key can decrypt the encrypted messages. -To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption). +To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption.md). We include the relevant configuration information of `CryptoConfig` into `ProducerConfig`. The specific configurable field information about `CryptoConfig` is as follows: diff --git a/site2/website/versioned_docs/version-2.7.4/getting-started-docker.md b/site2/website/versioned_docs/version-2.7.4/getting-started-docker.md index b9f12f0b52688..4f20971d75330 100644 --- a/site2/website/versioned_docs/version-2.7.4/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.7.4/getting-started-docker.md @@ -50,8 +50,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -110,7 +110,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.7.4/getting-started-helm.md b/site2/website/versioned_docs/version-2.7.4/getting-started-helm.md index 65a9fc04fe568..440087c275c05 100644 --- a/site2/website/versioned_docs/version-2.7.4/getting-started-helm.md +++ b/site2/website/versioned_docs/version-2.7.4/getting-started-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.7.4/getting-started-pulsar.md b/site2/website/versioned_docs/version-2.7.4/getting-started-pulsar.md index 11c5e66cb62c2..752590f57b558 100644 --- a/site2/website/versioned_docs/version-2.7.4/getting-started-pulsar.md +++ b/site2/website/versioned_docs/version-2.7.4/getting-started-pulsar.md @@ -5,13 +5,13 @@ sidebar_label: "Pulsar 2.0" original_id: pulsar-2.0 --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.7.4/getting-started-standalone.md b/site2/website/versioned_docs/version-2.7.4/getting-started-standalone.md index 4dfcf27b54e62..e94bb4bb6b9c3 100644 --- a/site2/website/versioned_docs/version-2.7.4/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.7.4/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -59,7 +59,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.7.4/helm-install.md b/site2/website/versioned_docs/version-2.7.4/helm-install.md index edb12260f010e..1f4d5eb69d5dd 100644 --- a/site2/website/versioned_docs/version-2.7.4/helm-install.md +++ b/site2/website/versioned_docs/version-2.7.4/helm-install.md @@ -21,7 +21,7 @@ Before deploying Pulsar, you need to prepare your environment. ### Tools -Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools) on your computer. +Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools.md) on your computer. ## Cloud cluster preparation @@ -37,8 +37,8 @@ To create and connect to the Kubernetes cluster, follow the instructions: ## Pulsar deployment -Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy). +Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy.md). ## Pulsar upgrade -To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade). +To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade.md). diff --git a/site2/website/versioned_docs/version-2.7.4/helm-overview.md b/site2/website/versioned_docs/version-2.7.4/helm-overview.md index 9cde85e2d18d5..385d535e319b6 100644 --- a/site2/website/versioned_docs/version-2.7.4/helm-overview.md +++ b/site2/website/versioned_docs/version-2.7.4/helm-overview.md @@ -52,9 +52,9 @@ It includes support for: ## Pulsar Helm chart quick start -To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install). +This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install.md). ## Troubleshooting @@ -64,7 +64,7 @@ We have done our best to make these charts as seamless as possible. Occasionally The Apache Pulsar Helm chart contains all required dependencies. -If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm) for your first iteration. +If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm.md) for your first iteration. 1. [Preparation](helm-prepare.md) 2. [Deployment](helm-deploy.md) @@ -82,7 +82,7 @@ helm upgrade apache/pulsar -f pulsar.yaml ``` -For more detailed information, see [Upgrading](helm-upgrade). +For more detailed information, see [Upgrading](helm-upgrade.md). ## Uninstallation diff --git a/site2/website/versioned_docs/version-2.7.4/helm-prepare.md b/site2/website/versioned_docs/version-2.7.4/helm-prepare.md index 95d7e668f9eca..5e9f2f9ef4f68 100644 --- a/site2/website/versioned_docs/version-2.7.4/helm-prepare.md +++ b/site2/website/versioned_docs/version-2.7.4/helm-prepare.md @@ -89,4 +89,4 @@ PROJECT= USE_LOCAL_SSD=true LOCAL_SSD_COUNT= ## Next Steps -Continue with the [installation of the chart](helm-deploy) once you have the cluster up and running. +Continue with the [installation of the chart](helm-deploy.md) once you have the cluster up and running. diff --git a/site2/website/versioned_docs/version-2.7.4/helm-tools.md b/site2/website/versioned_docs/version-2.7.4/helm-tools.md index 6894608b03473..a7e4017b4fc4f 100644 --- a/site2/website/versioned_docs/version-2.7.4/helm-tools.md +++ b/site2/website/versioned_docs/version-2.7.4/helm-tools.md @@ -25,7 +25,7 @@ You can get Helm from the project's [releases page](https://github.com/helm/helm ### Next steps -Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare). +Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare.md). ## Additional information diff --git a/site2/website/versioned_docs/version-2.7.4/helm-upgrade.md b/site2/website/versioned_docs/version-2.7.4/helm-upgrade.md index e39cf97407355..7d671e6bfb3c1 100644 --- a/site2/website/versioned_docs/version-2.7.4/helm-upgrade.md +++ b/site2/website/versioned_docs/version-2.7.4/helm-upgrade.md @@ -20,7 +20,7 @@ You can retrieve your previous `--set` arguments cleanly, with `helm get values To upgrade Apache Pulsar to a newer version, follow these steps: 1. Check the change log for the specific version you would like to upgrade to. -2. Go through [deployment documentation](helm-deploy) step by step. +2. Go through [deployment documentation](helm-deploy.md) step by step. 3. Extract your previous `--set` arguments with the following command. ```bash diff --git a/site2/website/versioned_docs/version-2.7.4/io-cdc.md b/site2/website/versioned_docs/version-2.7.4/io-cdc.md index 20f16ae0d6904..e6e662884826d 100644 --- a/site2/website/versioned_docs/version-2.7.4/io-cdc.md +++ b/site2/website/versioned_docs/version-2.7.4/io-cdc.md @@ -13,8 +13,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.7.4/io-develop.md b/site2/website/versioned_docs/version-2.7.4/io-develop.md index 4a815b3b9d6f6..7a356662560c9 100644 --- a/site2/website/versioned_docs/version-2.7.4/io-develop.md +++ b/site2/website/versioned_docs/version-2.7.4/io-develop.md @@ -8,15 +8,15 @@ original_id: io-develop This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -163,7 +163,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -192,7 +192,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). diff --git a/site2/website/versioned_docs/version-2.7.4/io-overview.md b/site2/website/versioned_docs/version-2.7.4/io-overview.md index 68df2472d6334..b538604edc46f 100644 --- a/site2/website/versioned_docs/version-2.7.4/io-overview.md +++ b/site2/website/versioned_docs/version-2.7.4/io-overview.md @@ -158,7 +158,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.7.4/io-quickstart.md b/site2/website/versioned_docs/version-2.7.4/io-quickstart.md index b8fea1560bdeb..8474c93f51336 100644 --- a/site2/website/versioned_docs/version-2.7.4/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.7.4/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you are able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you are able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However, all the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -109,7 +109,7 @@ This section demonstrates how to connect Pulsar to Cassandra. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink.md). ::: @@ -236,11 +236,11 @@ You can create a configuration file through one of the following methods. ``` -For more information, see [Cassandra sink connector](io-cassandra-sink). +For more information, see [Cassandra sink connector](io-cassandra-sink.md). ### Create a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on them. Run the following command to create a Cassandra sink connector with sink type _cassandra_ and the config file _examples/cassandra-sink.yml_ created previously. @@ -267,7 +267,7 @@ as a Pulsar Function and writes the messages produced in the topic _test_cassand ### Inspect a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * Get the information of a Cassandra sink. @@ -431,7 +431,7 @@ to monitor a connector and perform other operations on it. ### Delete a Cassandra Sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. ```bash @@ -454,7 +454,7 @@ This section demonstrates how to connect Pulsar to PostgreSQL. ::: ->For more information, see [JDBC sink connector](io-jdbc-sink). +>For more information, see [JDBC sink connector](io-jdbc-sink.md). ### Setup a PostgreSQL cluster @@ -614,7 +614,7 @@ In this section, you need to configure a JDBC sink connector. ### Create a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on it. This example creates a sink connector and specifies the desired information. @@ -660,7 +660,7 @@ The sink has been created successfully if the following message appears. ### Inspect a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * List all running JDBC sink(s). @@ -780,7 +780,7 @@ to monitor a connector and perform other operations on it. ### Stop a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to stop a connector and perform other operations on it. ```bash @@ -808,7 +808,7 @@ The sink instance has been stopped successfully if the following message disappe ### Restart a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to restart a connector and perform other operations on it. ```bash @@ -844,7 +844,7 @@ Note that `pulsar-admin sinks localrun options` **runs a sink connector locally* ### Update a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to update a connector and perform other operations on it. This example updates the parallelism of the _pulsar-postgres-jdbc-sink_ sink connector to 2. @@ -912,7 +912,7 @@ The result shows that the parallelism is 2. ### Delete a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. This example deletes the _pulsar-postgres-jdbc-sink_ sink connector. diff --git a/site2/website/versioned_docs/version-2.7.4/io-use.md b/site2/website/versioned_docs/version-2.7.4/io-use.md index b6409d4420ad7..da9ed746c4d37 100644 --- a/site2/website/versioned_docs/version-2.7.4/io-use.md +++ b/site2/website/versioned_docs/version-2.7.4/io-use.md @@ -15,7 +15,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.7.4/kubernetes-helm.md b/site2/website/versioned_docs/version-2.7.4/kubernetes-helm.md index a9f768e6a96b3..ea92a0968cd7d 100644 --- a/site2/website/versioned_docs/version-2.7.4/kubernetes-helm.md +++ b/site2/website/versioned_docs/version-2.7.4/kubernetes-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.7.4/reference-cli-tools.md b/site2/website/versioned_docs/version-2.7.4/reference-cli-tools.md index d17bd3efd9790..2bc3cc56130f5 100644 --- a/site2/website/versioned_docs/version-2.7.4/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.7.4/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.7.4/schema-get-started.md b/site2/website/versioned_docs/version-2.7.4/schema-get-started.md index 97e270586a26d..afacb0fa51f2e 100644 --- a/site2/website/versioned_docs/version-2.7.4/schema-get-started.md +++ b/site2/website/versioned_docs/version-2.7.4/schema-get-started.md @@ -17,7 +17,7 @@ Applications typically adopt one of the following approaches to guarantee type s #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ### Client-side approach diff --git a/site2/website/versioned_docs/version-2.7.4/schema-manage.md b/site2/website/versioned_docs/version-2.7.4/schema-manage.md index 98105408fb37d..5984fdbeaaa5f 100644 --- a/site2/website/versioned_docs/version-2.7.4/schema-manage.md +++ b/site2/website/versioned_docs/version-2.7.4/schema-manage.md @@ -160,7 +160,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.7.4/schema-understand.md b/site2/website/versioned_docs/version-2.7.4/schema-understand.md index 333c902b58319..d17c6168da59f 100644 --- a/site2/website/versioned_docs/version-2.7.4/schema-understand.md +++ b/site2/website/versioned_docs/version-2.7.4/schema-understand.md @@ -380,7 +380,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```java diff --git a/site2/website/versioned_docs/version-2.7.4/security-athenz.md b/site2/website/versioned_docs/version-2.7.4/security-athenz.md index 947c3f470be46..8a39fe25316d0 100644 --- a/site2/website/versioned_docs/version-2.7.4/security-athenz.md +++ b/site2/website/versioned_docs/version-2.7.4/security-athenz.md @@ -76,7 +76,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.7.4/security-authorization.md b/site2/website/versioned_docs/version-2.7.4/security-authorization.md index e6785873f0e10..7ac09b6f439ea 100644 --- a/site2/website/versioned_docs/version-2.7.4/security-authorization.md +++ b/site2/website/versioned_docs/version-2.7.4/security-authorization.md @@ -27,7 +27,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -59,7 +59,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -87,7 +87,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication diff --git a/site2/website/versioned_docs/version-2.7.4/security-bouncy-castle.md b/site2/website/versioned_docs/version-2.7.4/security-bouncy-castle.md index abc25c71b8ae2..be937055d8e31 100644 --- a/site2/website/versioned_docs/version-2.7.4/security-bouncy-castle.md +++ b/site2/website/versioned_docs/version-2.7.4/security-bouncy-castle.md @@ -16,7 +16,7 @@ In Pulsar, security and crypto have dependencies on BouncyCastle Jars. For the d `Bouncy Castle` provides both [FIPS](https://www.bouncycastle.org/fips_faq.html) and non-FIPS version. But in a JVM, you can not include both of the 2 versions, and you need to exclude the current version before include the other. -In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. +In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption.md). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. ## How BouncyCastle modules packaged in Pulsar diff --git a/site2/website/versioned_docs/version-2.7.4/security-extending.md b/site2/website/versioned_docs/version-2.7.4/security-extending.md index c088e3a2c1c2c..e7484453b8beb 100644 --- a/site2/website/versioned_docs/version-2.7.4/security-extending.md +++ b/site2/website/versioned_docs/version-2.7.4/security-extending.md @@ -9,7 +9,7 @@ Pulsar provides a way to use custom authentication and authorization mechanisms. ## Authentication -Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview). +Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview.md). You can use a custom authentication mechanism by providing the implementation in the form of two plugins. One plugin is for the Client library and the other plugin is for the Pulsar Proxy and/or Pulsar Broker to validate the credentials. diff --git a/site2/website/versioned_docs/version-2.7.4/security-jwt.md b/site2/website/versioned_docs/version-2.7.4/security-jwt.md index 656ab362f71b0..91117bedff753 100644 --- a/site2/website/versioned_docs/version-2.7.4/security-jwt.md +++ b/site2/website/versioned_docs/version-2.7.4/security-jwt.md @@ -32,11 +32,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -303,7 +303,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.7.4/security-kerberos.md b/site2/website/versioned_docs/version-2.7.4/security-kerberos.md index 670586a9d1691..c49fa3bea1fce 100644 --- a/site2/website/versioned_docs/version-2.7.4/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.7.4/security-kerberos.md @@ -373,7 +373,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.7.4/security-overview.md b/site2/website/versioned_docs/version-2.7.4/security-overview.md index c7823104d145b..a1664fcb8b450 100644 --- a/site2/website/versioned_docs/version-2.7.4/security-overview.md +++ b/site2/website/versioned_docs/version-2.7.4/security-overview.md @@ -21,7 +21,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.7.4/security-tls-authentication.md b/site2/website/versioned_docs/version-2.7.4/security-tls-authentication.md index 79aa8ea9fd13d..61c8aa43e1421 100644 --- a/site2/website/versioned_docs/version-2.7.4/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.7.4/security-tls-authentication.md @@ -7,9 +7,9 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -101,7 +101,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -121,7 +121,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.7.4/security-tls-keystore.md b/site2/website/versioned_docs/version-2.7.4/security-tls-keystore.md index 3ea385bc189fc..d0eed3992d5d0 100644 --- a/site2/website/versioned_docs/version-2.7.4/security-tls-keystore.md +++ b/site2/website/versioned_docs/version-2.7.4/security-tls-keystore.md @@ -7,7 +7,7 @@ original_id: security-tls-keystore ## Overview -Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication) between clients and Apache Pulsar service. +Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication.md) between clients and Apache Pulsar service. By default it uses PEM format file configuration. This page tries to describe use [KeyStore](https://en.wikipedia.org/wiki/Java_KeyStore) type configure for TLS. @@ -185,7 +185,7 @@ This is similar to [TLS encryption configuing for client with PEM type](security For a a minimal configuration, user need to provide the TrustStore information. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties @@ -278,7 +278,7 @@ webSocketServiceEnabled=false Besides the TLS encryption configuring. The main work is configuring the KeyStore, which contains a valid CN as client role, for client. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties diff --git a/site2/website/versioned_docs/version-2.7.4/security-tls-transport.md b/site2/website/versioned_docs/version-2.7.4/security-tls-transport.md index 12c4f2e22812b..31dd72589067e 100644 --- a/site2/website/versioned_docs/version-2.7.4/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.7.4/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,9 +27,9 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). -`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ## Create TLS certificates @@ -130,7 +130,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Broker Configuration -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -201,7 +201,7 @@ The examples below show that hostname verification is disabled for the CLI tools ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.7.4/security-token-admin.md b/site2/website/versioned_docs/version-2.7.4/security-token-admin.md index 2173a6cb1d11d..a5d69da9f0602 100644 --- a/site2/website/versioned_docs/version-2.7.4/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.7.4/security-token-admin.md @@ -163,7 +163,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.7.4/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.7.4/sql-deployment-configurations.md index c6d7c88a07e94..cc8bdbcd4fd51 100644 --- a/site2/website/versioned_docs/version-2.7.4/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.7.4/sql-deployment-configurations.md @@ -64,7 +64,7 @@ Since Pulsar SQL is powered by [Presto](https://prestosql.io), the configuration :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.7.4/sql-overview.md b/site2/website/versioned_docs/version-2.7.4/sql-overview.md index 3303923c6d237..e4dee1284869b 100644 --- a/site2/website/versioned_docs/version-2.7.4/sql-overview.md +++ b/site2/website/versioned_docs/version-2.7.4/sql-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Overview" original_id: sql-overview --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.7.4/standalone-docker.md b/site2/website/versioned_docs/version-2.7.4/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.7.4/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.7.4/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.7.4/standalone.md b/site2/website/versioned_docs/version-2.7.4/standalone.md index d8dd985f30d84..fdec1cc3d2bf3 100644 --- a/site2/website/versioned_docs/version-2.7.4/standalone.md +++ b/site2/website/versioned_docs/version-2.7.4/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -59,7 +59,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.7.4/tiered-storage-overview.md b/site2/website/versioned_docs/version-2.7.4/tiered-storage-overview.md index 7c6528ce2b36c..c635034f463b4 100644 --- a/site2/website/versioned_docs/version-2.7.4/tiered-storage-overview.md +++ b/site2/website/versioned_docs/version-2.7.4/tiered-storage-overview.md @@ -13,9 +13,9 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws). + - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws.md). - - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs). + - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs.md). ::: @@ -25,7 +25,7 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem). + For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem.md). ::: diff --git a/site2/website/versioned_docs/version-2.7.5/admin-api-brokers.md b/site2/website/versioned_docs/version-2.7.5/admin-api-brokers.md index 751bc14983205..61047c4d631d1 100644 --- a/site2/website/versioned_docs/version-2.7.5/admin-api-brokers.md +++ b/site2/website/versioned_docs/version-2.7.5/admin-api-brokers.md @@ -18,7 +18,7 @@ Pulsar brokers consist of two components: [Brokers](reference-terminology.md#broker) can be managed via: -* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/brokers` endpoint of the admin {@inject: rest:REST:/} API * The `brokers` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin.html} object in the [Java API](client-libraries-java.md) @@ -123,7 +123,7 @@ One way to configure a Pulsar [broker](reference-terminology.md#broker) is to su But since all broker configuration in Pulsar is stored in ZooKeeper, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values. -* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. +* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. * In the Pulsar admin {@inject: rest:REST:/} API, dynamic configuration is managed through the `/admin/v2/brokers/configuration` endpoint. ### Update dynamic configuration diff --git a/site2/website/versioned_docs/version-2.7.5/admin-api-clusters.md b/site2/website/versioned_docs/version-2.7.5/admin-api-clusters.md index f3af175295514..79508763e1db0 100644 --- a/site2/website/versioned_docs/version-2.7.5/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.7.5/admin-api-clusters.md @@ -16,7 +16,7 @@ servers (aka [bookies](reference-terminology.md#bookie)), and a [ZooKeeper](http Clusters can be managed via: -* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/clusters` endpoint of the admin {@inject: rest:REST:/} API * The `clusters` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md) @@ -79,7 +79,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -103,11 +103,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block -If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: +If you have [authentication](security-overview.md#authentication-providers) enabled, you will need to provide an auth configuration to use the [`pulsar-admin`](reference-pulsar-admin.md) tool. By default, the configuration for the `pulsar-admin` tool is found in the [`conf/client.conf`](reference-configuration.md#client) file. Here are the available parameters: |Name|Description|Default| |----|-----------|-------| diff --git a/site2/website/versioned_docs/version-2.7.5/admin-api-partitioned-topics.md b/site2/website/versioned_docs/version-2.7.5/admin-api-partitioned-topics.md index 28338e8cc4a69..4582ba14772d8 100644 --- a/site2/website/versioned_docs/version-2.7.5/admin-api-partitioned-topics.md +++ b/site2/website/versioned_docs/version-2.7.5/admin-api-partitioned-topics.md @@ -5,4 +5,4 @@ sidebar_label: "Partitioned topics" original_id: admin-api-partitioned-topics --- -For details of the content, refer to [manage topics](admin-api-topics). +For details of the content, refer to [manage topics](admin-api-topics.md). diff --git a/site2/website/versioned_docs/version-2.7.5/admin-api-permissions.md b/site2/website/versioned_docs/version-2.7.5/admin-api-permissions.md index 17f3f6ade8ba9..0931905ff8762 100644 --- a/site2/website/versioned_docs/version-2.7.5/admin-api-permissions.md +++ b/site2/website/versioned_docs/version-2.7.5/admin-api-permissions.md @@ -102,7 +102,7 @@ You can see which permissions have been granted to which roles in a namespace. values={[{"label":"pulsar-admin","value":"pulsar-admin"},{"label":"REST API","value":"REST API"},{"label":"Java","value":"Java"}]}> -Use the [`permissions`](reference-pulsar-admin#permissions) subcommand and specify a namespace: +Use the [`permissions`](reference-pulsar-admin#permissions.md) subcommand and specify a namespace: ```shell diff --git a/site2/website/versioned_docs/version-2.7.5/admin-api-persistent-topics.md b/site2/website/versioned_docs/version-2.7.5/admin-api-persistent-topics.md index 596e0358227ea..13e24f1d9eba4 100644 --- a/site2/website/versioned_docs/version-2.7.5/admin-api-persistent-topics.md +++ b/site2/website/versioned_docs/version-2.7.5/admin-api-persistent-topics.md @@ -5,4 +5,4 @@ sidebar_label: "Persistent topics" original_id: admin-api-persistent-topics --- -For details of the content, refer to [manage topics](admin-api-topics). +For details of the content, refer to [manage topics](admin-api-topics.md). diff --git a/site2/website/versioned_docs/version-2.7.5/admin-api-tenants.md b/site2/website/versioned_docs/version-2.7.5/admin-api-tenants.md index 63afe58620c62..d78aa2e55f4c3 100644 --- a/site2/website/versioned_docs/version-2.7.5/admin-api-tenants.md +++ b/site2/website/versioned_docs/version-2.7.5/admin-api-tenants.md @@ -11,7 +11,7 @@ import TabItem from '@theme/TabItem'; ```` -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -108,7 +108,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.7.5/client-libraries-cgo.md b/site2/website/versioned_docs/version-2.7.5/client-libraries-cgo.md index c79f7bb965473..f352f942b7714 100644 --- a/site2/website/versioned_docs/version-2.7.5/client-libraries-cgo.md +++ b/site2/website/versioned_docs/version-2.7.5/client-libraries-cgo.md @@ -24,7 +24,7 @@ Currently, the following Go clients are maintained in two repositories. ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Install go package @@ -57,7 +57,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -75,7 +75,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -236,14 +236,14 @@ Parameter | Description | Default `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | `Properties`| Attach a set of application defined properties to the producer. This properties will be visible in the topic stats | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | Set the number of max pending messages across all the partitions. This setting will be used to lower the max pending messages for each partition `MaxPendingMessages(int)`, if the total exceeds the configured value.| `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | `Batching` | Control whether automatic batching of messages is enabled for the producer. | false `BatchingMaxPublishDelay` | Set the time period within which the messages sent will be batched (default: 1ms) if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or until | 1ms `BatchingMaxMessages` | Set the maximum number of messages permitted in a batch. (default: 1000) If set to a value greater than 1, messages will be queued until this threshold is reached or batch interval has elapsed | 1000 @@ -512,7 +512,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.7.5/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.7.5/client-libraries-cpp.md index bb52d9effa404..684df5fd2d825 100644 --- a/site2/website/versioned_docs/version-2.7.5/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.7.5/client-libraries-cpp.md @@ -261,7 +261,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema -This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started). +This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started.md). ### Create producer with Avro schema diff --git a/site2/website/versioned_docs/version-2.7.5/client-libraries-dotnet.md b/site2/website/versioned_docs/version-2.7.5/client-libraries-dotnet.md index ade664cd95277..b574fa0b2e5ed 100644 --- a/site2/website/versioned_docs/version-2.7.5/client-libraries-dotnet.md +++ b/site2/website/versioned_docs/version-2.7.5/client-libraries-dotnet.md @@ -170,7 +170,7 @@ var client = PulsarClient.Builder() Currently, the Pulsar C# client supports the TLS (Transport Layer Security) and JWT (JSON Web Token) authentication. -If you have followed [Authentication using TLS](security-tls-authentication), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: +If you have followed [Authentication using TLS](security-tls-authentication.md), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: 1. Create an unencrypted and password-less pfx file. diff --git a/site2/website/versioned_docs/version-2.7.5/client-libraries-go.md b/site2/website/versioned_docs/version-2.7.5/client-libraries-go.md index 97705a643c049..6c05e1d733422 100644 --- a/site2/website/versioned_docs/version-2.7.5/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.7.5/client-libraries-go.md @@ -35,7 +35,7 @@ import "github.com/apache/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -53,7 +53,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -680,7 +680,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker @@ -700,7 +700,7 @@ opts := pulsar.ClientOptions{ ## OAuth2 authentication -To use [OAuth2 authentication](security-oauth2), you'll need to configure your client to perform the following operations. +To use [OAuth2 authentication](security-oauth2.md), you'll need to configure your client to perform the following operations. This example shows how to configure OAuth2 authentication. ```go diff --git a/site2/website/versioned_docs/version-2.7.5/client-libraries-java.md b/site2/website/versioned_docs/version-2.7.5/client-libraries-java.md index bbe9473622fbe..17f01bf1647f8 100644 --- a/site2/website/versioned_docs/version-2.7.5/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.7.5/client-libraries-java.md @@ -5,7 +5,7 @@ sidebar_label: "Java" original_id: client-libraries-java --- -You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), and [reader](#reader) of a Java client are thread-safe. @@ -14,10 +14,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Includes both `pulsar-client` and `pulsar-client-admin`

    Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.

    In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -57,7 +57,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -83,7 +83,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -114,7 +114,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -237,7 +237,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -535,7 +535,7 @@ private void receiveMessageFromConsumer(Object consumer) { ### Subscription types -Pulsar has various [subscription types](concepts-messaging#subscription-types) to match different scenarios. A topic can have multiple subscriptions with different subscription types. However, a subscription can only have one subscription type at a time. +Pulsar has various [subscription types](concepts-messaging.md#subscription-types) to match different scenarios. A topic can have multiple subscriptions with different subscription types. However, a subscription can only have one subscription type at a time. A subscription is identical with the subscription name; a subscription name can specify only one subscription type at a time. To change the subscription type, you should first stop all consumers of this subscription. @@ -822,7 +822,7 @@ Total hash range size is 65536, so the max end of the range should be less than ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -936,7 +936,7 @@ The following schema formats are currently available for Java: ## Authentication -Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2). You can use the Pulsar Java client with all of them. +Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2.md). You can use the Pulsar Java client with all of them. ### TLS Authentication @@ -963,7 +963,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` @@ -1000,7 +1000,7 @@ PulsarClient client = PulsarClient.builder() ### Oauth2 -The following example shows how to use [Oauth2](security-oauth2) as an authentication provider for the Pulsar Java client. +The following example shows how to use [Oauth2](security-oauth2.md) as an authentication provider for the Pulsar Java client. You can use the factory method to configure authentication for Pulsar Java client. diff --git a/site2/website/versioned_docs/version-2.7.5/client-libraries-node.md b/site2/website/versioned_docs/version-2.7.5/client-libraries-node.md index 12c1bdf590f1a..db851088a571f 100644 --- a/site2/website/versioned_docs/version-2.7.5/client-libraries-node.md +++ b/site2/website/versioned_docs/version-2.7.5/client-libraries-node.md @@ -46,7 +46,7 @@ Also, this library works only in Node.js 10.x or later because it uses the [`nod ::: ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here is an example for `localhost`: @@ -64,7 +64,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication), the URL will look like something like this: +If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication.md), the URL will look like something like this: ```http @@ -99,7 +99,7 @@ The following configurable parameters are available for Pulsar clients: | Parameter | Description | Default | | :-------- | :---------- | :------ | | `serviceUrl` | The connection URL for the Pulsar cluster. See [above](#connection-urls) for more info. | | -| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication) for more info. | | +| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication.md) for more info. | | | `operationTimeoutSeconds` | The timeout for Node.js client operations (creating producers, subscribing to and unsubscribing from [topics](reference-terminology.md#topic)). Retries will occur until this threshold is reached, at which point the operation will fail. | 30 | | `ioThreads` | The number of threads to use for handling connections to Pulsar [brokers](reference-terminology.md#broker). | 1 | | `messageListenerThreads` | The number of threads used by message listeners ([consumers](#consumers) and [readers](#readers)). | 1 | @@ -150,7 +150,7 @@ Pulsar Node.js producers have the following methods available: | :-------- | :---------- | :------ | | `topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer publishes messages. The topic format is `` or `//`. For example, `sample/ns1/my-topic`. | | | `producerName` | A name for the producer. If you do not explicitly assign a name, Pulsar will automatically generate a globally unique name. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | | -| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30000 | +| `sendTimeoutMs` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `sendTimeoutMs` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30000 | | `initialSequenceId` | The initial sequence ID of the message. When producer send message, add sequence ID to message. The ID is increased each time to send. | | | `maxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `send` method will fail *unless* `blockIfQueueFull` is set to `true`. | 1000 | | `maxPendingMessagesAcrossPartitions` | The maximum size of the sum of partition's pending queue. | 50000 | diff --git a/site2/website/versioned_docs/version-2.7.5/client-libraries-python.md b/site2/website/versioned_docs/version-2.7.5/client-libraries-python.md index 7d44db909ca9f..e7ea7f01813f8 100644 --- a/site2/website/versioned_docs/version-2.7.5/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.7.5/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. diff --git a/site2/website/versioned_docs/version-2.7.5/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.7.5/client-libraries-websocket.md index cdf87d70aa905..4906a93ec0e6f 100644 --- a/site2/website/versioned_docs/version-2.7.5/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.7.5/client-libraries-websocket.md @@ -5,14 +5,14 @@ sidebar_label: "WebSocket" original_id: client-libraries-websocket --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.7.5/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.7.5/concepts-architecture-overview.md index c65f9149ceda5..ad3f04858b25a 100644 --- a/site2/website/versioned_docs/version-2.7.5/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.7.5/concepts-architecture-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Architecture" original_id: concepts-architecture-overview --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -17,20 +17,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -40,9 +40,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -130,17 +130,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -148,7 +148,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.7.5/concepts-authentication.md b/site2/website/versioned_docs/version-2.7.5/concepts-authentication.md index b375ecb16a2d8..f6307890c904a 100644 --- a/site2/website/versioned_docs/version-2.7.5/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.7.5/concepts-authentication.md @@ -5,5 +5,5 @@ sidebar_label: "Authentication and Authorization" original_id: concepts-authentication --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.7.5/concepts-clients.md b/site2/website/versioned_docs/version-2.7.5/concepts-clients.md index b68f76a2d8b08..4040624f7d636 100644 --- a/site2/website/versioned_docs/version-2.7.5/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.7.5/concepts-clients.md @@ -5,12 +5,12 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -38,7 +38,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.7.5/concepts-messaging.md b/site2/website/versioned_docs/version-2.7.5/concepts-messaging.md index a4ea0637c94eb..4b03824755273 100644 --- a/site2/website/versioned_docs/version-2.7.5/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.7.5/concepts-messaging.md @@ -21,8 +21,8 @@ Messages are the basic "unit" of Pulsar. The following table lists the component Component | Description :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer who produces the message. If you do not specify a producer name, the default name is used. Sequence ID | Each Pulsar message belongs to an ordered sequence on its topic. The sequence ID of the message is its order in that sequence. @@ -30,7 +30,7 @@ Publish time | The timestamp of when the message is published. The timestamp is Event time | An optional timestamp attached to a message by applications. For example, applications attach a timestamp on when the message is processed. If nothing is set to event time, the value is `0`. TypedMessageBuilder | It is used to construct a message. You can set message properties such as the message key, message value with `TypedMessageBuilder`.
    When you set `TypedMessageBuilder`, set the key as a string. If you set the key as other types, for example, an AVRO object, the key is sent as bytes, and it is difficult to get the AVRO object back on the consumer. -> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol). +> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -107,7 +107,7 @@ Messages are received from [brokers](reference-terminology.md#broker) either syn ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -344,7 +344,7 @@ A [subscription](#subscriptions) can have one or more consumers. When a consumer #### When to use -By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry). +By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry.md). #### How to use @@ -448,7 +448,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -460,7 +460,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -498,7 +498,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -553,7 +553,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -576,12 +576,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication.md). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.7.5/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.7.5/concepts-multi-tenancy.md index be752ccbbf9ca..c4acb21acb743 100644 --- a/site2/website/versioned_docs/version-2.7.5/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.7.5/concepts-multi-tenancy.md @@ -5,7 +5,7 @@ sidebar_label: "Multi Tenancy" original_id: concepts-multi-tenancy --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -21,7 +21,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -29,7 +29,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.7.5/concepts-overview.md b/site2/website/versioned_docs/version-2.7.5/concepts-overview.md index c1262b38daf16..e8a2f4b9d321a 100644 --- a/site2/website/versioned_docs/version-2.7.5/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.7.5/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.7.5/concepts-replication.md b/site2/website/versioned_docs/version-2.7.5/concepts-replication.md index 6e23962e99771..799f0eb4d92c6 100644 --- a/site2/website/versioned_docs/version-2.7.5/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.7.5/concepts-replication.md @@ -5,5 +5,5 @@ sidebar_label: "Geo Replication" original_id: concepts-replication --- -Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that. +Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo.md) in Pulsar enables you to do that. diff --git a/site2/website/versioned_docs/version-2.7.5/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.7.5/concepts-tiered-storage.md index 0b45b0a2ed501..f6988e53a8cd4 100644 --- a/site2/website/versioned_docs/version-2.7.5/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.7.5/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.7.5/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.7.5/concepts-topic-compaction.md index e402c965384fd..34b7ed7fbbd31 100644 --- a/site2/website/versioned_docs/version-2.7.5/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.7.5/concepts-topic-compaction.md @@ -7,7 +7,7 @@ original_id: concepts-topic-compaction Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -24,7 +24,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.7.5/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.7.5/cookbooks-compaction.md index 6fcbc197f9ecb..dfa314727241a 100644 --- a/site2/website/versioned_docs/version-2.7.5/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.7.5/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.7.5/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.7.5/cookbooks-non-persistent.md index 391569a8cbc6b..178301e86eb8d 100644 --- a/site2/website/versioned_docs/version-2.7.5/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.7.5/cookbooks-non-persistent.md @@ -41,7 +41,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -55,7 +55,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.7.5/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.7.5/cookbooks-partitioned.md index 39de6de952689..f279f2265f666 100644 --- a/site2/website/versioned_docs/version-2.7.5/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.7.5/cookbooks-partitioned.md @@ -4,4 +4,4 @@ title: Partitioned topics sidebar_label: "Partitioned Topics" original_id: cookbooks-partitioned --- -For details of the content, refer to [manage topics](admin-api-topics). +For details of the content, refer to [manage topics](admin-api-topics.md). diff --git a/site2/website/versioned_docs/version-2.7.5/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.7.5/cookbooks-retention-expiry.md index 5295b4c5de4ae..23175d607b37e 100644 --- a/site2/website/versioned_docs/version-2.7.5/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.7.5/cookbooks-retention-expiry.md @@ -22,7 +22,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.7.5/deploy-aws.md b/site2/website/versioned_docs/version-2.7.5/deploy-aws.md index 1c8f1a08d7249..662e99ceb9b09 100644 --- a/site2/website/versioned_docs/version-2.7.5/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.7.5/deploy-aws.md @@ -5,7 +5,7 @@ sidebar_label: "Amazon Web Services" original_id: deploy-aws --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -210,7 +210,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.7.5/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.7.5/deploy-bare-metal-multi-cluster.md index 4ee6da37f0a90..74303a3ff7924 100644 --- a/site2/website/versioned_docs/version-2.7.5/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.7.5/deploy-bare-metal-multi-cluster.md @@ -9,17 +9,17 @@ original_id: deploy-bare-metal-multi-cluster 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using it in a startup or on a single team, you had better opt for a single cluster. For instructions on deploying a single cluster, -see the guide [here](deploy-bare-metal). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo). Deploying a multi-cluster Pulsar instance involves the following basic steps: +A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md). Deploying a multi-cluster Pulsar instance involves the following basic steps: * Deploying two separate [ZooKeeper](#deploy-zookeeper) quorums: a [local](#deploy-local-zookeeper) quorum for each cluster in the instance and a [configuration store](#configuration-store) quorum for instance-wide tasks * Initializing [cluster metadata](#cluster-metadata-initialization) for each cluster @@ -29,7 +29,7 @@ A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You If you want to deploy a single Pulsar cluster, see [Clusters and Brokers](getting-started-standalone.md#start-the-cluster). > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). ## System requirement Pulsar is currently available for **MacOS** and **Linux**. In order to use Pulsar, you need to install Java 8 from [Oracle download center](http://www.oracle.com/). @@ -67,7 +67,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -245,7 +245,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -364,17 +364,17 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). -You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > #### Service discovery already provided by many scheduling systems -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ### Service discovery setup -The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol) of Pulsar. +The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol.md) of Pulsar. To get started setting up the built-in service of discovery of Pulsar, you need to change a few parameters in the [`conf/discovery.conf`](reference-configuration.md#service-discovery) configuration file. Set the [`zookeeperServers`](reference-configuration.md#service-discovery-zookeeperServers) parameter to the ZooKeeper quorum connection string of the cluster and the [`configurationStoreServers`](reference-configuration.md#service-discovery-configurationStoreServers) setting to the [configuration store](reference-terminology.md#configuration-store) quorum connection string. @@ -399,7 +399,7 @@ $ bin/pulsar-daemon start discovery ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.7.5/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.7.5/deploy-bare-metal.md index 7b1850b3b21aa..1ea6286f6e533 100644 --- a/site2/website/versioned_docs/version-2.7.5/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.7.5/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using Pulsar in a startup or on a single team, it is simplest to opt for a single cluster. If you do need to run a multi-cluster Pulsar instance, -see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -99,7 +99,7 @@ The extracted directory contains the following subdirectories: Directory | Contains :---------|:-------- -`bin` |[command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` |[command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `data` | The data storage directory that ZooKeeper and BookKeeper use `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -261,9 +261,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (you had better not use a different port). -`--web-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). +`--web-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should not use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (you had better not use a different port). -`--broker-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). +`--broker-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). > If you do not have a DNS server, you can use multi-host format in the service URL with the following settings: @@ -403,7 +403,7 @@ webServicePortTls=8443 ### Enable Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -421,7 +421,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker). +If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker.md). ### Start Brokers diff --git a/site2/website/versioned_docs/version-2.7.5/deploy-dcos.md b/site2/website/versioned_docs/version-2.7.5/deploy-dcos.md index f5f8d1faa7dbe..952d5f47e30fa 100644 --- a/site2/website/versioned_docs/version-2.7.5/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.7.5/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -If you want to enable all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of +If you want to enable all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.7.5/deploy-docker.md b/site2/website/versioned_docs/version-2.7.5/deploy-docker.md index 64a0939aa787b..8348d78deb237 100644 --- a/site2/website/versioned_docs/version-2.7.5/deploy-docker.md +++ b/site2/website/versioned_docs/version-2.7.5/deploy-docker.md @@ -57,4 +57,4 @@ docker network connect pulsar broker To check whether the containers are successfully connected to the network, enter the `docker network inspect pulsar` command. -For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal). +For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal.md). diff --git a/site2/website/versioned_docs/version-2.7.5/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.7.5/deploy-kubernetes.md index 248d840c3016d..12b92350e03e8 100644 --- a/site2/website/versioned_docs/version-2.7.5/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.7.5/deploy-kubernetes.md @@ -6,6 +6,6 @@ original_id: deploy-kubernetes --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). diff --git a/site2/website/versioned_docs/version-2.7.5/deploy-monitoring.md b/site2/website/versioned_docs/version-2.7.5/deploy-monitoring.md index 4b4f0b40f4007..214502a7e8fea 100644 --- a/site2/website/versioned_docs/version-2.7.5/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.7.5/deploy-monitoring.md @@ -86,7 +86,7 @@ Those metrics are added in the Prometheus interface, you can monitor and check t You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy). +When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy.md). ## Dashboards @@ -94,7 +94,7 @@ When you collect time series statistics, the major problem is to make sure the n ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager). +The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.7.5/develop-schema.md b/site2/website/versioned_docs/version-2.7.5/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.7.5/develop-schema.md +++ b/site2/website/versioned_docs/version-2.7.5/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.7.5/functions-deploy.md b/site2/website/versioned_docs/version-2.7.5/functions-deploy.md index 51f1140693dbe..d9496b3ed5b48 100644 --- a/site2/website/versioned_docs/version-2.7.5/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.7.5/functions-deploy.md @@ -9,12 +9,12 @@ original_id: functions-deploy To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface diff --git a/site2/website/versioned_docs/version-2.7.5/functions-develop.md b/site2/website/versioned_docs/version-2.7.5/functions-develop.md index 8c7e828153979..0911074c5381f 100644 --- a/site2/website/versioned_docs/version-2.7.5/functions-develop.md +++ b/site2/website/versioned_docs/version-2.7.5/functions-develop.md @@ -1163,7 +1163,7 @@ public class MetricRecorderFunction implements Function { ``` -> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring) guide. +> For instructions on reading and using metrics, see the [Monitoring](deploy-monitoring.md) guide.     @@ -1194,11 +1194,11 @@ Currently, the feature is not available in Go. ```` ### Access metrics -To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring) in Pulsar. +To access metrics created by Pulsar Functions, refer to [Monitoring](deploy-monitoring.md) in Pulsar. ## Security -If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker). For more details, refer to [Security settings](functions-worker.md#security-settings). +If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings). Pulsar Functions can support the following providers: diff --git a/site2/website/versioned_docs/version-2.7.5/functions-overview.md b/site2/website/versioned_docs/version-2.7.5/functions-overview.md index 25dc6029acf70..816d301e0fd0e 100644 --- a/site2/website/versioned_docs/version-2.7.5/functions-overview.md +++ b/site2/website/versioned_docs/version-2.7.5/functions-overview.md @@ -164,7 +164,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.7.5/functions-package.md b/site2/website/versioned_docs/version-2.7.5/functions-package.md index d7a5b3ec20b60..ac25cb255e110 100644 --- a/site2/website/versioned_docs/version-2.7.5/functions-package.md +++ b/site2/website/versioned_docs/version-2.7.5/functions-package.md @@ -15,7 +15,7 @@ Currently, the window function is not available in Python and Go. ## Prerequisite -Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm). +Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm.md). To check whether the Docker image starts, you can use the `docker ps` command. diff --git a/site2/website/versioned_docs/version-2.7.5/functions-worker.md b/site2/website/versioned_docs/version-2.7.5/functions-worker.md index fa1467aa5fd30..9b603e88de609 100644 --- a/site2/website/versioned_docs/version-2.7.5/functions-worker.md +++ b/site2/website/versioned_docs/version-2.7.5/functions-worker.md @@ -4,7 +4,7 @@ title: Deploy and manage functions worker sidebar_label: "Setup: Pulsar Functions Worker" original_id: functions-worker --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -180,7 +180,7 @@ brokerClientTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -200,7 +200,7 @@ authenticationProviders: [ provider1, provider2 ] ``` For *TLS Authentication* provider, follow the example below to add the necessary settings. -See [TLS Authentication](security-tls-authentication) for more details. +See [TLS Authentication](security-tls-authentication.md) for more details. ``` @@ -224,7 +224,7 @@ properties: ``` For *Token Authentication* provider, add necessary settings for `properties` if needed. -See [Token Authentication](security-jwt) for more details. +See [Token Authentication](security-jwt.md) for more details. ``` @@ -262,7 +262,7 @@ superUserRoles: You can use the public and private key pair that the application configures to perform encryption. Only the consumers with a valid key can decrypt the encrypted messages. -To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption). +To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption.md). We include the relevant configuration information of `CryptoConfig` into `ProducerConfig`. The specific configurable field information about `CryptoConfig` is as follows: diff --git a/site2/website/versioned_docs/version-2.7.5/getting-started-docker.md b/site2/website/versioned_docs/version-2.7.5/getting-started-docker.md index b9f12f0b52688..4f20971d75330 100644 --- a/site2/website/versioned_docs/version-2.7.5/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.7.5/getting-started-docker.md @@ -50,8 +50,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -110,7 +110,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.7.5/getting-started-helm.md b/site2/website/versioned_docs/version-2.7.5/getting-started-helm.md index 65a9fc04fe568..440087c275c05 100644 --- a/site2/website/versioned_docs/version-2.7.5/getting-started-helm.md +++ b/site2/website/versioned_docs/version-2.7.5/getting-started-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.7.5/getting-started-pulsar.md b/site2/website/versioned_docs/version-2.7.5/getting-started-pulsar.md index 11c5e66cb62c2..752590f57b558 100644 --- a/site2/website/versioned_docs/version-2.7.5/getting-started-pulsar.md +++ b/site2/website/versioned_docs/version-2.7.5/getting-started-pulsar.md @@ -5,13 +5,13 @@ sidebar_label: "Pulsar 2.0" original_id: pulsar-2.0 --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.7.5/getting-started-standalone.md b/site2/website/versioned_docs/version-2.7.5/getting-started-standalone.md index 4dfcf27b54e62..e94bb4bb6b9c3 100644 --- a/site2/website/versioned_docs/version-2.7.5/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.7.5/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -59,7 +59,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.7.5/helm-install.md b/site2/website/versioned_docs/version-2.7.5/helm-install.md index edb12260f010e..1f4d5eb69d5dd 100644 --- a/site2/website/versioned_docs/version-2.7.5/helm-install.md +++ b/site2/website/versioned_docs/version-2.7.5/helm-install.md @@ -21,7 +21,7 @@ Before deploying Pulsar, you need to prepare your environment. ### Tools -Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools) on your computer. +Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools.md) on your computer. ## Cloud cluster preparation @@ -37,8 +37,8 @@ To create and connect to the Kubernetes cluster, follow the instructions: ## Pulsar deployment -Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy). +Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy.md). ## Pulsar upgrade -To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade). +To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade.md). diff --git a/site2/website/versioned_docs/version-2.7.5/helm-overview.md b/site2/website/versioned_docs/version-2.7.5/helm-overview.md index 9cde85e2d18d5..385d535e319b6 100644 --- a/site2/website/versioned_docs/version-2.7.5/helm-overview.md +++ b/site2/website/versioned_docs/version-2.7.5/helm-overview.md @@ -52,9 +52,9 @@ It includes support for: ## Pulsar Helm chart quick start -To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install). +This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install.md). ## Troubleshooting @@ -64,7 +64,7 @@ We have done our best to make these charts as seamless as possible. Occasionally The Apache Pulsar Helm chart contains all required dependencies. -If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm) for your first iteration. +If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm.md) for your first iteration. 1. [Preparation](helm-prepare.md) 2. [Deployment](helm-deploy.md) @@ -82,7 +82,7 @@ helm upgrade apache/pulsar -f pulsar.yaml ``` -For more detailed information, see [Upgrading](helm-upgrade). +For more detailed information, see [Upgrading](helm-upgrade.md). ## Uninstallation diff --git a/site2/website/versioned_docs/version-2.7.5/helm-prepare.md b/site2/website/versioned_docs/version-2.7.5/helm-prepare.md index 95d7e668f9eca..5e9f2f9ef4f68 100644 --- a/site2/website/versioned_docs/version-2.7.5/helm-prepare.md +++ b/site2/website/versioned_docs/version-2.7.5/helm-prepare.md @@ -89,4 +89,4 @@ PROJECT= USE_LOCAL_SSD=true LOCAL_SSD_COUNT= ## Next Steps -Continue with the [installation of the chart](helm-deploy) once you have the cluster up and running. +Continue with the [installation of the chart](helm-deploy.md) once you have the cluster up and running. diff --git a/site2/website/versioned_docs/version-2.7.5/helm-tools.md b/site2/website/versioned_docs/version-2.7.5/helm-tools.md index 6894608b03473..a7e4017b4fc4f 100644 --- a/site2/website/versioned_docs/version-2.7.5/helm-tools.md +++ b/site2/website/versioned_docs/version-2.7.5/helm-tools.md @@ -25,7 +25,7 @@ You can get Helm from the project's [releases page](https://github.com/helm/helm ### Next steps -Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare). +Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare.md). ## Additional information diff --git a/site2/website/versioned_docs/version-2.7.5/helm-upgrade.md b/site2/website/versioned_docs/version-2.7.5/helm-upgrade.md index e39cf97407355..7d671e6bfb3c1 100644 --- a/site2/website/versioned_docs/version-2.7.5/helm-upgrade.md +++ b/site2/website/versioned_docs/version-2.7.5/helm-upgrade.md @@ -20,7 +20,7 @@ You can retrieve your previous `--set` arguments cleanly, with `helm get values To upgrade Apache Pulsar to a newer version, follow these steps: 1. Check the change log for the specific version you would like to upgrade to. -2. Go through [deployment documentation](helm-deploy) step by step. +2. Go through [deployment documentation](helm-deploy.md) step by step. 3. Extract your previous `--set` arguments with the following command. ```bash diff --git a/site2/website/versioned_docs/version-2.7.5/io-cdc.md b/site2/website/versioned_docs/version-2.7.5/io-cdc.md index 20f16ae0d6904..e6e662884826d 100644 --- a/site2/website/versioned_docs/version-2.7.5/io-cdc.md +++ b/site2/website/versioned_docs/version-2.7.5/io-cdc.md @@ -13,8 +13,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.7.5/io-develop.md b/site2/website/versioned_docs/version-2.7.5/io-develop.md index 4a815b3b9d6f6..7a356662560c9 100644 --- a/site2/website/versioned_docs/version-2.7.5/io-develop.md +++ b/site2/website/versioned_docs/version-2.7.5/io-develop.md @@ -8,15 +8,15 @@ original_id: io-develop This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -163,7 +163,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -192,7 +192,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). diff --git a/site2/website/versioned_docs/version-2.7.5/io-overview.md b/site2/website/versioned_docs/version-2.7.5/io-overview.md index 68df2472d6334..b538604edc46f 100644 --- a/site2/website/versioned_docs/version-2.7.5/io-overview.md +++ b/site2/website/versioned_docs/version-2.7.5/io-overview.md @@ -158,7 +158,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](reference-connector-admin.md#sources) and [sinks](reference-connector-admin.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.7.5/io-quickstart.md b/site2/website/versioned_docs/version-2.7.5/io-quickstart.md index b8fea1560bdeb..8474c93f51336 100644 --- a/site2/website/versioned_docs/version-2.7.5/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.7.5/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you are able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you are able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However, all the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -109,7 +109,7 @@ This section demonstrates how to connect Pulsar to Cassandra. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink.md). ::: @@ -236,11 +236,11 @@ You can create a configuration file through one of the following methods. ``` -For more information, see [Cassandra sink connector](io-cassandra-sink). +For more information, see [Cassandra sink connector](io-cassandra-sink.md). ### Create a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on them. Run the following command to create a Cassandra sink connector with sink type _cassandra_ and the config file _examples/cassandra-sink.yml_ created previously. @@ -267,7 +267,7 @@ as a Pulsar Function and writes the messages produced in the topic _test_cassand ### Inspect a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * Get the information of a Cassandra sink. @@ -431,7 +431,7 @@ to monitor a connector and perform other operations on it. ### Delete a Cassandra Sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. ```bash @@ -454,7 +454,7 @@ This section demonstrates how to connect Pulsar to PostgreSQL. ::: ->For more information, see [JDBC sink connector](io-jdbc-sink). +>For more information, see [JDBC sink connector](io-jdbc-sink.md). ### Setup a PostgreSQL cluster @@ -614,7 +614,7 @@ In this section, you need to configure a JDBC sink connector. ### Create a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on it. This example creates a sink connector and specifies the desired information. @@ -660,7 +660,7 @@ The sink has been created successfully if the following message appears. ### Inspect a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * List all running JDBC sink(s). @@ -780,7 +780,7 @@ to monitor a connector and perform other operations on it. ### Stop a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to stop a connector and perform other operations on it. ```bash @@ -808,7 +808,7 @@ The sink instance has been stopped successfully if the following message disappe ### Restart a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to restart a connector and perform other operations on it. ```bash @@ -844,7 +844,7 @@ Note that `pulsar-admin sinks localrun options` **runs a sink connector locally* ### Update a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to update a connector and perform other operations on it. This example updates the parallelism of the _pulsar-postgres-jdbc-sink_ sink connector to 2. @@ -912,7 +912,7 @@ The result shows that the parallelism is 2. ### Delete a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. This example deletes the _pulsar-postgres-jdbc-sink_ sink connector. diff --git a/site2/website/versioned_docs/version-2.7.5/io-use.md b/site2/website/versioned_docs/version-2.7.5/io-use.md index b6409d4420ad7..da9ed746c4d37 100644 --- a/site2/website/versioned_docs/version-2.7.5/io-use.md +++ b/site2/website/versioned_docs/version-2.7.5/io-use.md @@ -15,7 +15,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.7.5/kubernetes-helm.md b/site2/website/versioned_docs/version-2.7.5/kubernetes-helm.md index a9f768e6a96b3..ea92a0968cd7d 100644 --- a/site2/website/versioned_docs/version-2.7.5/kubernetes-helm.md +++ b/site2/website/versioned_docs/version-2.7.5/kubernetes-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.7.5/reference-cli-tools.md b/site2/website/versioned_docs/version-2.7.5/reference-cli-tools.md index d17bd3efd9790..2bc3cc56130f5 100644 --- a/site2/website/versioned_docs/version-2.7.5/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.7.5/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.7.5/schema-get-started.md b/site2/website/versioned_docs/version-2.7.5/schema-get-started.md index 97e270586a26d..afacb0fa51f2e 100644 --- a/site2/website/versioned_docs/version-2.7.5/schema-get-started.md +++ b/site2/website/versioned_docs/version-2.7.5/schema-get-started.md @@ -17,7 +17,7 @@ Applications typically adopt one of the following approaches to guarantee type s #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ### Client-side approach diff --git a/site2/website/versioned_docs/version-2.7.5/schema-manage.md b/site2/website/versioned_docs/version-2.7.5/schema-manage.md index 98105408fb37d..5984fdbeaaa5f 100644 --- a/site2/website/versioned_docs/version-2.7.5/schema-manage.md +++ b/site2/website/versioned_docs/version-2.7.5/schema-manage.md @@ -160,7 +160,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.7.5/schema-understand.md b/site2/website/versioned_docs/version-2.7.5/schema-understand.md index 333c902b58319..d17c6168da59f 100644 --- a/site2/website/versioned_docs/version-2.7.5/schema-understand.md +++ b/site2/website/versioned_docs/version-2.7.5/schema-understand.md @@ -380,7 +380,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```java diff --git a/site2/website/versioned_docs/version-2.7.5/security-athenz.md b/site2/website/versioned_docs/version-2.7.5/security-athenz.md index 947c3f470be46..8a39fe25316d0 100644 --- a/site2/website/versioned_docs/version-2.7.5/security-athenz.md +++ b/site2/website/versioned_docs/version-2.7.5/security-athenz.md @@ -76,7 +76,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.7.5/security-authorization.md b/site2/website/versioned_docs/version-2.7.5/security-authorization.md index e6785873f0e10..7ac09b6f439ea 100644 --- a/site2/website/versioned_docs/version-2.7.5/security-authorization.md +++ b/site2/website/versioned_docs/version-2.7.5/security-authorization.md @@ -27,7 +27,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -59,7 +59,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -87,7 +87,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication diff --git a/site2/website/versioned_docs/version-2.7.5/security-bouncy-castle.md b/site2/website/versioned_docs/version-2.7.5/security-bouncy-castle.md index abc25c71b8ae2..be937055d8e31 100644 --- a/site2/website/versioned_docs/version-2.7.5/security-bouncy-castle.md +++ b/site2/website/versioned_docs/version-2.7.5/security-bouncy-castle.md @@ -16,7 +16,7 @@ In Pulsar, security and crypto have dependencies on BouncyCastle Jars. For the d `Bouncy Castle` provides both [FIPS](https://www.bouncycastle.org/fips_faq.html) and non-FIPS version. But in a JVM, you can not include both of the 2 versions, and you need to exclude the current version before include the other. -In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. +In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption.md). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. ## How BouncyCastle modules packaged in Pulsar diff --git a/site2/website/versioned_docs/version-2.7.5/security-extending.md b/site2/website/versioned_docs/version-2.7.5/security-extending.md index c088e3a2c1c2c..e7484453b8beb 100644 --- a/site2/website/versioned_docs/version-2.7.5/security-extending.md +++ b/site2/website/versioned_docs/version-2.7.5/security-extending.md @@ -9,7 +9,7 @@ Pulsar provides a way to use custom authentication and authorization mechanisms. ## Authentication -Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview). +Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview.md). You can use a custom authentication mechanism by providing the implementation in the form of two plugins. One plugin is for the Client library and the other plugin is for the Pulsar Proxy and/or Pulsar Broker to validate the credentials. diff --git a/site2/website/versioned_docs/version-2.7.5/security-jwt.md b/site2/website/versioned_docs/version-2.7.5/security-jwt.md index 656ab362f71b0..91117bedff753 100644 --- a/site2/website/versioned_docs/version-2.7.5/security-jwt.md +++ b/site2/website/versioned_docs/version-2.7.5/security-jwt.md @@ -32,11 +32,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -303,7 +303,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.7.5/security-kerberos.md b/site2/website/versioned_docs/version-2.7.5/security-kerberos.md index 670586a9d1691..c49fa3bea1fce 100644 --- a/site2/website/versioned_docs/version-2.7.5/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.7.5/security-kerberos.md @@ -373,7 +373,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.7.5/security-overview.md b/site2/website/versioned_docs/version-2.7.5/security-overview.md index c7823104d145b..a1664fcb8b450 100644 --- a/site2/website/versioned_docs/version-2.7.5/security-overview.md +++ b/site2/website/versioned_docs/version-2.7.5/security-overview.md @@ -21,7 +21,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.7.5/security-tls-authentication.md b/site2/website/versioned_docs/version-2.7.5/security-tls-authentication.md index 79aa8ea9fd13d..61c8aa43e1421 100644 --- a/site2/website/versioned_docs/version-2.7.5/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.7.5/security-tls-authentication.md @@ -7,9 +7,9 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -101,7 +101,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -121,7 +121,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.7.5/security-tls-keystore.md b/site2/website/versioned_docs/version-2.7.5/security-tls-keystore.md index 3ea385bc189fc..d0eed3992d5d0 100644 --- a/site2/website/versioned_docs/version-2.7.5/security-tls-keystore.md +++ b/site2/website/versioned_docs/version-2.7.5/security-tls-keystore.md @@ -7,7 +7,7 @@ original_id: security-tls-keystore ## Overview -Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication) between clients and Apache Pulsar service. +Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication.md) between clients and Apache Pulsar service. By default it uses PEM format file configuration. This page tries to describe use [KeyStore](https://en.wikipedia.org/wiki/Java_KeyStore) type configure for TLS. @@ -185,7 +185,7 @@ This is similar to [TLS encryption configuing for client with PEM type](security For a a minimal configuration, user need to provide the TrustStore information. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties @@ -278,7 +278,7 @@ webSocketServiceEnabled=false Besides the TLS encryption configuring. The main work is configuring the KeyStore, which contains a valid CN as client role, for client. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties diff --git a/site2/website/versioned_docs/version-2.7.5/security-tls-transport.md b/site2/website/versioned_docs/version-2.7.5/security-tls-transport.md index 12c4f2e22812b..31dd72589067e 100644 --- a/site2/website/versioned_docs/version-2.7.5/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.7.5/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,9 +27,9 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). -`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ## Create TLS certificates @@ -130,7 +130,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Broker Configuration -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -201,7 +201,7 @@ The examples below show that hostname verification is disabled for the CLI tools ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.7.5/security-token-admin.md b/site2/website/versioned_docs/version-2.7.5/security-token-admin.md index 2173a6cb1d11d..a5d69da9f0602 100644 --- a/site2/website/versioned_docs/version-2.7.5/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.7.5/security-token-admin.md @@ -163,7 +163,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.7.5/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.7.5/sql-deployment-configurations.md index c6d7c88a07e94..cc8bdbcd4fd51 100644 --- a/site2/website/versioned_docs/version-2.7.5/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.7.5/sql-deployment-configurations.md @@ -64,7 +64,7 @@ Since Pulsar SQL is powered by [Presto](https://prestosql.io), the configuration :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.7.5/sql-overview.md b/site2/website/versioned_docs/version-2.7.5/sql-overview.md index 3303923c6d237..e4dee1284869b 100644 --- a/site2/website/versioned_docs/version-2.7.5/sql-overview.md +++ b/site2/website/versioned_docs/version-2.7.5/sql-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Overview" original_id: sql-overview --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Presto](https://prestosql.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.7.5/standalone-docker.md b/site2/website/versioned_docs/version-2.7.5/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.7.5/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.7.5/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.7.5/standalone.md b/site2/website/versioned_docs/version-2.7.5/standalone.md index d8dd985f30d84..fdec1cc3d2bf3 100644 --- a/site2/website/versioned_docs/version-2.7.5/standalone.md +++ b/site2/website/versioned_docs/version-2.7.5/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -59,7 +59,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -68,7 +68,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -117,7 +117,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -131,7 +131,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -164,12 +164,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -202,7 +202,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.7.5/tiered-storage-overview.md b/site2/website/versioned_docs/version-2.7.5/tiered-storage-overview.md index 7c6528ce2b36c..c635034f463b4 100644 --- a/site2/website/versioned_docs/version-2.7.5/tiered-storage-overview.md +++ b/site2/website/versioned_docs/version-2.7.5/tiered-storage-overview.md @@ -13,9 +13,9 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws). + - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws.md). - - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs). + - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs.md). ::: @@ -25,7 +25,7 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem). + For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem.md). ::: diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/admin-api-brokers.md b/site2/website/versioned_docs/version-2.8.0-deprecated/admin-api-brokers.md index d51a71e160366..4af4363850efd 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/admin-api-brokers.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/admin-api-brokers.md @@ -18,7 +18,7 @@ Pulsar brokers consist of two components: [Brokers](reference-terminology.md#broker) can be managed via: -* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/brokers` endpoint of the admin {@inject: rest:REST:/} API * The `brokers` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin.html} object in the [Java API](client-libraries-java.md) @@ -166,7 +166,7 @@ One way to configure a Pulsar [broker](reference-terminology.md#broker) is to su But since all broker configuration in Pulsar is stored in ZooKeeper, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values. -* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. +* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. * In the Pulsar admin {@inject: rest:REST:/} API, dynamic configuration is managed through the `/admin/v2/brokers/configuration` endpoint. ### Update dynamic configuration diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/admin-api-clusters.md b/site2/website/versioned_docs/version-2.8.0-deprecated/admin-api-clusters.md index 3eb238506cc69..e0e9fb5f91f65 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/admin-api-clusters.md @@ -16,7 +16,7 @@ servers (aka [bookies](reference-terminology.md#bookie)), and a [ZooKeeper](http Clusters can be managed via: -* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/clusters` endpoint of the admin {@inject: rest:REST:/} API * The `clusters` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md) @@ -79,7 +79,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -103,11 +103,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-cgo.md b/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-cgo.md index c79f7bb965473..f352f942b7714 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-cgo.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-cgo.md @@ -24,7 +24,7 @@ Currently, the following Go clients are maintained in two repositories. ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Install go package @@ -57,7 +57,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -75,7 +75,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -236,14 +236,14 @@ Parameter | Description | Default `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | `Properties`| Attach a set of application defined properties to the producer. This properties will be visible in the topic stats | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | Set the number of max pending messages across all the partitions. This setting will be used to lower the max pending messages for each partition `MaxPendingMessages(int)`, if the total exceeds the configured value.| `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | `Batching` | Control whether automatic batching of messages is enabled for the producer. | false `BatchingMaxPublishDelay` | Set the time period within which the messages sent will be batched (default: 1ms) if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or until | 1ms `BatchingMaxMessages` | Set the maximum number of messages permitted in a batch. (default: 1000) If set to a value greater than 1, messages will be queued until this threshold is reached or batch interval has elapsed | 1000 @@ -512,7 +512,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-cpp.md index deebe13bada6d..08622b9e83071 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-cpp.md @@ -372,7 +372,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema -This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started). +This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started.md). ### Create producer with Avro schema diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-dotnet.md b/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-dotnet.md index ade664cd95277..b574fa0b2e5ed 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-dotnet.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-dotnet.md @@ -170,7 +170,7 @@ var client = PulsarClient.Builder() Currently, the Pulsar C# client supports the TLS (Transport Layer Security) and JWT (JSON Web Token) authentication. -If you have followed [Authentication using TLS](security-tls-authentication), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: +If you have followed [Authentication using TLS](security-tls-authentication.md), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: 1. Create an unencrypted and password-less pfx file. diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-go.md b/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-go.md index fbb2573aa5db8..6281b03dd8c80 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-go.md @@ -35,7 +35,7 @@ import "github.com/apache/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -61,7 +61,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -844,7 +844,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker @@ -864,7 +864,7 @@ opts := pulsar.ClientOptions{ ## OAuth2 authentication -To use [OAuth2 authentication](security-oauth2), you'll need to configure your client to perform the following operations. +To use [OAuth2 authentication](security-oauth2.md), you'll need to configure your client to perform the following operations. This example shows how to configure OAuth2 authentication. ```go diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-java.md b/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-java.md index 9c4e129e3e683..5ee69dd5a8e7c 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-java.md @@ -5,7 +5,7 @@ sidebar_label: "Java" original_id: client-libraries-java --- -You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), and [reader](#reader) of a Java client are thread-safe. @@ -14,10 +14,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Includes both `pulsar-client` and `pulsar-client-admin`

    Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.

    In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -57,7 +57,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -83,7 +83,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -114,7 +114,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -237,7 +237,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -824,7 +824,7 @@ Total hash range size is 65536, so the max end of the range should be less than ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -942,7 +942,7 @@ For example of ProtobufNativeSchema, see [`SchemaDefinition` in `Complex type`]( ## Authentication -Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2). You can use the Pulsar Java client with all of them. +Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2.md). You can use the Pulsar Java client with all of them. ### TLS Authentication @@ -969,7 +969,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` @@ -1006,7 +1006,7 @@ PulsarClient client = PulsarClient.builder() ### Oauth2 -The following example shows how to use [Oauth2](security-oauth2) as an authentication provider for the Pulsar Java client. +The following example shows how to use [Oauth2](security-oauth2.md) as an authentication provider for the Pulsar Java client. You can use the factory method to configure authentication for Pulsar Java client. diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-node.md b/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-node.md index 20ec2432dd38e..e24032946bdcd 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-node.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-node.md @@ -48,7 +48,7 @@ Also, this library works only in Node.js 10.x or later because it uses the [`nod ::: ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here is an example for `localhost`: @@ -66,7 +66,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication), the URL looks like this: +If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication.md), the URL looks like this: ```http @@ -101,7 +101,7 @@ The following configurable parameters are available for Pulsar clients: | Parameter | Description | Default | | :-------- | :---------- | :------ | | `serviceUrl` | The connection URL for the Pulsar cluster. See [above](#connection-urls) for more info. | | -| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication) for more info. | | +| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication.md) for more info. | | | `operationTimeoutSeconds` | The timeout for Node.js client operations (creating producers, subscribing to and unsubscribing from [topics](reference-terminology.md#topic)). Retries occur until this threshold is reached, at which point the operation fails. | 30 | | `ioThreads` | The number of threads to use for handling connections to Pulsar [brokers](reference-terminology.md#broker). | 1 | | `messageListenerThreads` | The number of threads used by message listeners ([consumers](#consumers) and [readers](#readers)). | 1 | @@ -154,7 +154,7 @@ Pulsar Node.js producers have the following methods available: | :-------- | :---------- | :------ | | `topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer publishes messages. The topic format is `` or `//`. For example, `sample/ns1/my-topic`. | | | `producerName` | A name for the producer. If you do not explicitly assign a name, Pulsar automatically generates a globally unique name. If you choose to explicitly assign a name, it needs to be unique across *all* Pulsar clusters, otherwise the creation operation throws an error. | | -| `sendTimeoutMs` | When publishing a message to a topic, the producer waits for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error is thrown. If you set `sendTimeoutMs` to -1, the timeout is set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30000 | +| `sendTimeoutMs` | When publishing a message to a topic, the producer waits for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error is thrown. If you set `sendTimeoutMs` to -1, the timeout is set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30000 | | `initialSequenceId` | The initial sequence ID of the message. When producer send message, add sequence ID to message. The ID is increased each time to send. | | | `maxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `send` method fails *unless* `blockIfQueueFull` is set to `true`. | 1000 | | `maxPendingMessagesAcrossPartitions` | The maximum size of the sum of partition's pending queue. | 50000 | diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-python.md b/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-python.md index 7ebc6bec64e5d..f30cf55387d92 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-websocket.md index bc13b437a97c1..ebdb9bc1cd18f 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/client-libraries-websocket.md @@ -5,14 +5,14 @@ sidebar_label: "WebSocket" original_id: client-libraries-websocket --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-architecture-overview.md index 0c7d44260ddeb..f3e75c3e307e0 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-architecture-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Architecture" original_id: concepts-architecture-overview --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -17,20 +17,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers. The producers connect to the brokers to publish messages and the consumers connect to the brokers to consume the messages. -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -40,9 +40,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -136,17 +136,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -154,7 +154,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-authentication.md b/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-authentication.md index b375ecb16a2d8..f6307890c904a 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-authentication.md @@ -5,5 +5,5 @@ sidebar_label: "Authentication and Authorization" original_id: concepts-authentication --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-clients.md b/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-clients.md index b68f76a2d8b08..4040624f7d636 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-clients.md @@ -5,12 +5,12 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -38,7 +38,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-messaging.md b/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-messaging.md index 76732ef5002ac..c2a545d7f8d51 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-messaging.md @@ -21,8 +21,8 @@ Messages are the basic "unit" of Pulsar. The following table lists the component Component | Description :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer who produces the message. If you do not specify a producer name, the default name is used. Sequence ID | Each Pulsar message belongs to an ordered sequence on its topic. The sequence ID of the message is its order in that sequence. @@ -50,7 +50,7 @@ The default size of a message is 5 MB. You can configure the max size of a messa ``` -> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol). +> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -147,7 +147,7 @@ Messages are received from [brokers](reference-terminology.md#broker) either syn ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -384,7 +384,7 @@ A [subscription](#subscriptions) can have one or more consumers. When a consumer #### When to use -By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry). +By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry.md). #### How to use @@ -488,7 +488,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -500,7 +500,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -538,7 +538,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -625,7 +625,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -648,12 +648,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication.md). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-multi-tenancy.md index 730458c904618..93a59557b2efc 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-multi-tenancy.md @@ -5,7 +5,7 @@ sidebar_label: "Multi Tenancy" original_id: concepts-multi-tenancy --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -21,7 +21,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -29,7 +29,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-overview.md b/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-overview.md index c1262b38daf16..e8a2f4b9d321a 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-replication.md b/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-replication.md index 6e23962e99771..799f0eb4d92c6 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-replication.md @@ -5,5 +5,5 @@ sidebar_label: "Geo Replication" original_id: concepts-replication --- -Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that. +Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo.md) in Pulsar enables you to do that. diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-tiered-storage.md index 0b45b0a2ed501..f6988e53a8cd4 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-topic-compaction.md index e402c965384fd..34b7ed7fbbd31 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/concepts-topic-compaction.md @@ -7,7 +7,7 @@ original_id: concepts-topic-compaction Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -24,7 +24,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.8.0-deprecated/cookbooks-compaction.md index 6fcbc197f9ecb..dfa314727241a 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.8.0-deprecated/cookbooks-non-persistent.md index 391569a8cbc6b..178301e86eb8d 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/cookbooks-non-persistent.md @@ -41,7 +41,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -55,7 +55,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.8.0-deprecated/cookbooks-partitioned.md index 7882fb9025312..fb9ac354cc6d6 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/cookbooks-partitioned.md @@ -4,4 +4,4 @@ title: Partitioned topics sidebar_label: "Partitioned Topics" original_id: cookbooks-partitioned --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.8.0-deprecated/cookbooks-retention-expiry.md index af4be6bb2d9c6..192fbe6c53277 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/cookbooks-retention-expiry.md @@ -22,7 +22,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-aws.md b/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-aws.md index 071effeffde24..93c389b56e2cf 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-aws.md @@ -5,7 +5,7 @@ sidebar_label: "Amazon Web Services" original_id: deploy-aws --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -210,7 +210,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors) , edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md) , edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-bare-metal-multi-cluster.md index dacd571365aa2..1b23eea07a20b 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-bare-metal-multi-cluster.md @@ -9,17 +9,17 @@ original_id: deploy-bare-metal-multi-cluster 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using it in a startup or on a single team, you had better opt for a single cluster. For instructions on deploying a single cluster, -see the guide [here](deploy-bare-metal). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo). Deploying a multi-cluster Pulsar instance involves the following basic steps: +A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md). Deploying a multi-cluster Pulsar instance involves the following basic steps: * Deploying two separate [ZooKeeper](#deploy-zookeeper) quorums: a [local](#deploy-local-zookeeper) quorum for each cluster in the instance and a [configuration store](#configuration-store) quorum for instance-wide tasks * Initializing [cluster metadata](#cluster-metadata-initialization) for each cluster @@ -29,7 +29,7 @@ A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You If you want to deploy a single Pulsar cluster, see [Clusters and Brokers](getting-started-standalone.md#start-the-cluster). > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). ## System requirement @@ -74,7 +74,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -252,7 +252,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -371,17 +371,17 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). -You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > #### Service discovery already provided by many scheduling systems -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ### Service discovery setup -The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol) of Pulsar. +The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol.md) of Pulsar. To get started setting up the built-in service of discovery of Pulsar, you need to change a few parameters in the [`conf/discovery.conf`](reference-configuration.md#service-discovery) configuration file. Set the [`zookeeperServers`](reference-configuration.md#service-discovery-zookeeperServers) parameter to the ZooKeeper quorum connection string of the cluster and the [`configurationStoreServers`](reference-configuration.md#service-discovery-configurationStoreServers) setting to the [configuration store](reference-terminology.md#configuration-store) quorum connection string. @@ -406,7 +406,7 @@ $ bin/pulsar-daemon start discovery ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-bare-metal.md index f856f9f8b1ca6..bdd05f24f2566 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using Pulsar in a startup or on a single team, it is simplest to opt for a single cluster. If you do need to run a multi-cluster Pulsar instance, -see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -107,7 +107,7 @@ The extracted directory contains the following subdirectories: Directory | Contains :---------|:-------- -`bin` |[command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` |[command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `data` | The data storage directory that ZooKeeper and BookKeeper use `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -261,9 +261,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (you had better not use a different port). -`--web-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). +`--web-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should not use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (you had better not use a different port). -`--broker-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). +`--broker-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). > If you do not have a DNS server, you can use multi-host format in the service URL with the following settings: @@ -403,7 +403,7 @@ webServicePortTls=8443 ### Enable Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -421,7 +421,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker). +If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker.md). ### Start Brokers diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-dcos.md b/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-dcos.md index f5f8d1faa7dbe..952d5f47e30fa 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -If you want to enable all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of +If you want to enable all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-docker.md b/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-docker.md index 64a0939aa787b..8348d78deb237 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-docker.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-docker.md @@ -57,4 +57,4 @@ docker network connect pulsar broker To check whether the containers are successfully connected to the network, enter the `docker network inspect pulsar` command. -For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal). +For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal.md). diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-kubernetes.md index dc7123d000467..1aefc6ad79f71 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-kubernetes.md @@ -6,6 +6,6 @@ original_id: deploy-kubernetes --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-monitoring.md b/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-monitoring.md index ae2ac29943331..f9fe0e0bb97be 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/deploy-monitoring.md @@ -113,7 +113,7 @@ http://$FUNCTIONS_WORKER_ADDRESS:$WORKER_PORT/metrics: You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy). +When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy.md). ## Dashboards @@ -121,7 +121,7 @@ When you collect time series statistics, the major problem is to make sure the n ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager). +The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/develop-schema.md b/site2/website/versioned_docs/version-2.8.0-deprecated/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/develop-schema.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/functions-deploy.md b/site2/website/versioned_docs/version-2.8.0-deprecated/functions-deploy.md index 51f1140693dbe..d9496b3ed5b48 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/functions-deploy.md @@ -9,12 +9,12 @@ original_id: functions-deploy To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/functions-develop.md b/site2/website/versioned_docs/version-2.8.0-deprecated/functions-develop.md index 36ea4b4a21073..2e29aa1c47400 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/functions-develop.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/functions-develop.md @@ -1144,7 +1144,7 @@ You can monitor Pulsar Functions that have been deployed with the following meth - Check the metrics provided by Pulsar. - Pulsar Functions expose the metrics that can be collected and used for monitoring the health of **Java, Python, and Go** functions. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar Functions expose the metrics that can be collected and used for monitoring the health of **Java, Python, and Go** functions. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. For the complete list of the function metrics, see [here](reference-metrics.md#pulsar-functions). @@ -1214,7 +1214,7 @@ Currently, the feature is not available in Go. ## Security -If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker). For more details, refer to [Security settings](functions-worker.md#security-settings). +If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings). Pulsar Functions can support the following providers: diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/functions-overview.md b/site2/website/versioned_docs/version-2.8.0-deprecated/functions-overview.md index 25dc6029acf70..816d301e0fd0e 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/functions-overview.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/functions-overview.md @@ -164,7 +164,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/functions-package.md b/site2/website/versioned_docs/version-2.8.0-deprecated/functions-package.md index 82346aa8c5098..a995d5c158877 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/functions-package.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/functions-package.md @@ -15,7 +15,7 @@ Currently, the window function is not available in Python and Go. ## Prerequisite -Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm). +Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm.md). To check whether the Docker image starts, you can use the `docker ps` command. diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/functions-worker.md b/site2/website/versioned_docs/version-2.8.0-deprecated/functions-worker.md index e8a3eec702b6a..1ad643cee8431 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/functions-worker.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/functions-worker.md @@ -4,7 +4,7 @@ title: Deploy and manage functions worker sidebar_label: "Setup: Pulsar Functions Worker" original_id: functions-worker --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -179,7 +179,7 @@ brokerClientTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -199,7 +199,7 @@ authenticationProviders: [ provider1, provider2 ] ``` For *TLS Authentication* provider, follow the example below to add the necessary settings. -See [TLS Authentication](security-tls-authentication) for more details. +See [TLS Authentication](security-tls-authentication.md) for more details. ``` @@ -223,7 +223,7 @@ properties: ``` For *Token Authentication* provider, add necessary settings for `properties` if needed. -See [Token Authentication](security-jwt) for more details. +See [Token Authentication](security-jwt.md) for more details. Note: key files must be DER-encoded ``` @@ -262,7 +262,7 @@ superUserRoles: You can use the public and private key pair that the application configures to perform encryption. Only the consumers with a valid key can decrypt the encrypted messages. -To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption). +To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption.md). We include the relevant configuration information of `CryptoConfig` into `ProducerConfig`. The specific configurable field information about `CryptoConfig` is as follows: diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/getting-started-docker.md b/site2/website/versioned_docs/version-2.8.0-deprecated/getting-started-docker.md index b9f12f0b52688..4f20971d75330 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/getting-started-docker.md @@ -50,8 +50,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -110,7 +110,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/getting-started-helm.md b/site2/website/versioned_docs/version-2.8.0-deprecated/getting-started-helm.md index 65a9fc04fe568..440087c275c05 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/getting-started-helm.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/getting-started-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/getting-started-pulsar.md b/site2/website/versioned_docs/version-2.8.0-deprecated/getting-started-pulsar.md index 11c5e66cb62c2..752590f57b558 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/getting-started-pulsar.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/getting-started-pulsar.md @@ -5,13 +5,13 @@ sidebar_label: "Pulsar 2.0" original_id: pulsar-2.0 --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/getting-started-standalone.md b/site2/website/versioned_docs/version-2.8.0-deprecated/getting-started-standalone.md index 0e40286d71dcb..cea47efd08d4b 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -65,7 +65,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -74,7 +74,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -123,7 +123,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -137,7 +137,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -170,12 +170,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -208,7 +208,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/helm-install.md b/site2/website/versioned_docs/version-2.8.0-deprecated/helm-install.md index edb12260f010e..1f4d5eb69d5dd 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/helm-install.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/helm-install.md @@ -21,7 +21,7 @@ Before deploying Pulsar, you need to prepare your environment. ### Tools -Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools) on your computer. +Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools.md) on your computer. ## Cloud cluster preparation @@ -37,8 +37,8 @@ To create and connect to the Kubernetes cluster, follow the instructions: ## Pulsar deployment -Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy). +Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy.md). ## Pulsar upgrade -To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade). +To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade.md). diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/helm-overview.md b/site2/website/versioned_docs/version-2.8.0-deprecated/helm-overview.md index 9cde85e2d18d5..385d535e319b6 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/helm-overview.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/helm-overview.md @@ -52,9 +52,9 @@ It includes support for: ## Pulsar Helm chart quick start -To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install). +This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install.md). ## Troubleshooting @@ -64,7 +64,7 @@ We have done our best to make these charts as seamless as possible. Occasionally The Apache Pulsar Helm chart contains all required dependencies. -If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm) for your first iteration. +If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm.md) for your first iteration. 1. [Preparation](helm-prepare.md) 2. [Deployment](helm-deploy.md) @@ -82,7 +82,7 @@ helm upgrade apache/pulsar -f pulsar.yaml ``` -For more detailed information, see [Upgrading](helm-upgrade). +For more detailed information, see [Upgrading](helm-upgrade.md). ## Uninstallation diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/helm-prepare.md b/site2/website/versioned_docs/version-2.8.0-deprecated/helm-prepare.md index 95d7e668f9eca..5e9f2f9ef4f68 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/helm-prepare.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/helm-prepare.md @@ -89,4 +89,4 @@ PROJECT= USE_LOCAL_SSD=true LOCAL_SSD_COUNT= ## Next Steps -Continue with the [installation of the chart](helm-deploy) once you have the cluster up and running. +Continue with the [installation of the chart](helm-deploy.md) once you have the cluster up and running. diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/helm-tools.md b/site2/website/versioned_docs/version-2.8.0-deprecated/helm-tools.md index efd61eae32506..6ba89006913b6 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/helm-tools.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/helm-tools.md @@ -25,7 +25,7 @@ You can get Helm from the project's [releases page](https://github.com/helm/helm ### Next steps -Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare). +Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare.md). ## Additional information diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/helm-upgrade.md b/site2/website/versioned_docs/version-2.8.0-deprecated/helm-upgrade.md index e39cf97407355..7d671e6bfb3c1 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/helm-upgrade.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/helm-upgrade.md @@ -20,7 +20,7 @@ You can retrieve your previous `--set` arguments cleanly, with `helm get values To upgrade Apache Pulsar to a newer version, follow these steps: 1. Check the change log for the specific version you would like to upgrade to. -2. Go through [deployment documentation](helm-deploy) step by step. +2. Go through [deployment documentation](helm-deploy.md) step by step. 3. Extract your previous `--set` arguments with the following command. ```bash diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/io-cdc.md b/site2/website/versioned_docs/version-2.8.0-deprecated/io-cdc.md index 20f16ae0d6904..e6e662884826d 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/io-cdc.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/io-cdc.md @@ -13,8 +13,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/io-develop.md b/site2/website/versioned_docs/version-2.8.0-deprecated/io-develop.md index a20206fbf6a06..d6f4f8261ac82 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/io-develop.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/io-develop.md @@ -14,15 +14,15 @@ import TabItem from '@theme/TabItem'; This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -275,7 +275,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -304,7 +304,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). @@ -379,7 +379,7 @@ Pulsar connectors enable you to move data in and out of Pulsar easily. It is imp - Check the metrics provided by Pulsar. - Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. - Set and check your customized metrics. diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/io-overview.md b/site2/website/versioned_docs/version-2.8.0-deprecated/io-overview.md index 0070a015296db..3db5ee34042d3 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/io-overview.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/io-overview.md @@ -158,7 +158,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/io-quickstart.md b/site2/website/versioned_docs/version-2.8.0-deprecated/io-quickstart.md index b8fea1560bdeb..8474c93f51336 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you are able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you are able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However, all the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -109,7 +109,7 @@ This section demonstrates how to connect Pulsar to Cassandra. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink.md). ::: @@ -236,11 +236,11 @@ You can create a configuration file through one of the following methods. ``` -For more information, see [Cassandra sink connector](io-cassandra-sink). +For more information, see [Cassandra sink connector](io-cassandra-sink.md). ### Create a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on them. Run the following command to create a Cassandra sink connector with sink type _cassandra_ and the config file _examples/cassandra-sink.yml_ created previously. @@ -267,7 +267,7 @@ as a Pulsar Function and writes the messages produced in the topic _test_cassand ### Inspect a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * Get the information of a Cassandra sink. @@ -431,7 +431,7 @@ to monitor a connector and perform other operations on it. ### Delete a Cassandra Sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. ```bash @@ -454,7 +454,7 @@ This section demonstrates how to connect Pulsar to PostgreSQL. ::: ->For more information, see [JDBC sink connector](io-jdbc-sink). +>For more information, see [JDBC sink connector](io-jdbc-sink.md). ### Setup a PostgreSQL cluster @@ -614,7 +614,7 @@ In this section, you need to configure a JDBC sink connector. ### Create a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on it. This example creates a sink connector and specifies the desired information. @@ -660,7 +660,7 @@ The sink has been created successfully if the following message appears. ### Inspect a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * List all running JDBC sink(s). @@ -780,7 +780,7 @@ to monitor a connector and perform other operations on it. ### Stop a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to stop a connector and perform other operations on it. ```bash @@ -808,7 +808,7 @@ The sink instance has been stopped successfully if the following message disappe ### Restart a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to restart a connector and perform other operations on it. ```bash @@ -844,7 +844,7 @@ Note that `pulsar-admin sinks localrun options` **runs a sink connector locally* ### Update a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to update a connector and perform other operations on it. This example updates the parallelism of the _pulsar-postgres-jdbc-sink_ sink connector to 2. @@ -912,7 +912,7 @@ The result shows that the parallelism is 2. ### Delete a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. This example deletes the _pulsar-postgres-jdbc-sink_ sink connector. diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/io-use.md b/site2/website/versioned_docs/version-2.8.0-deprecated/io-use.md index b6409d4420ad7..da9ed746c4d37 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/io-use.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/io-use.md @@ -15,7 +15,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/kubernetes-helm.md b/site2/website/versioned_docs/version-2.8.0-deprecated/kubernetes-helm.md index a9f768e6a96b3..ea92a0968cd7d 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/kubernetes-helm.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/kubernetes-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/reference-cli-tools.md b/site2/website/versioned_docs/version-2.8.0-deprecated/reference-cli-tools.md index ecd6363453e3c..309009f635e29 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/reference-configuration.md b/site2/website/versioned_docs/version-2.8.0-deprecated/reference-configuration.md index cbe7be52b498b..eb3bc1c574feb 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/reference-configuration.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/reference-configuration.md @@ -8,7 +8,7 @@ original_id: reference-configuration -You can manage Pulsar configuration by configuration files in the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a Pulsar [installation](getting-started-standalone). +You can manage Pulsar configuration by configuration files in the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a Pulsar [installation](getting-started-standalone.md). - [BookKeeper](#bookkeeper) - [Broker](#broker) diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/schema-get-started.md b/site2/website/versioned_docs/version-2.8.0-deprecated/schema-get-started.md index 97e270586a26d..afacb0fa51f2e 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/schema-get-started.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/schema-get-started.md @@ -17,7 +17,7 @@ Applications typically adopt one of the following approaches to guarantee type s #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ### Client-side approach diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/schema-manage.md b/site2/website/versioned_docs/version-2.8.0-deprecated/schema-manage.md index 4ae363882b2d5..c588aae619eee 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/schema-manage.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/schema-manage.md @@ -160,7 +160,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/schema-understand.md b/site2/website/versioned_docs/version-2.8.0-deprecated/schema-understand.md index 153d96e6e71cb..a86b02add435e 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/schema-understand.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/schema-understand.md @@ -459,7 +459,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```java diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/security-athenz.md b/site2/website/versioned_docs/version-2.8.0-deprecated/security-athenz.md index 947c3f470be46..8a39fe25316d0 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/security-athenz.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/security-athenz.md @@ -76,7 +76,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/security-authorization.md b/site2/website/versioned_docs/version-2.8.0-deprecated/security-authorization.md index e6785873f0e10..7ac09b6f439ea 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/security-authorization.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/security-authorization.md @@ -27,7 +27,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -59,7 +59,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -87,7 +87,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/security-bouncy-castle.md b/site2/website/versioned_docs/version-2.8.0-deprecated/security-bouncy-castle.md index abc25c71b8ae2..be937055d8e31 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/security-bouncy-castle.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/security-bouncy-castle.md @@ -16,7 +16,7 @@ In Pulsar, security and crypto have dependencies on BouncyCastle Jars. For the d `Bouncy Castle` provides both [FIPS](https://www.bouncycastle.org/fips_faq.html) and non-FIPS version. But in a JVM, you can not include both of the 2 versions, and you need to exclude the current version before include the other. -In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. +In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption.md). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. ## How BouncyCastle modules packaged in Pulsar diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/security-extending.md b/site2/website/versioned_docs/version-2.8.0-deprecated/security-extending.md index c088e3a2c1c2c..e7484453b8beb 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/security-extending.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/security-extending.md @@ -9,7 +9,7 @@ Pulsar provides a way to use custom authentication and authorization mechanisms. ## Authentication -Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview). +Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview.md). You can use a custom authentication mechanism by providing the implementation in the form of two plugins. One plugin is for the Client library and the other plugin is for the Pulsar Proxy and/or Pulsar Broker to validate the credentials. diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/security-jwt.md b/site2/website/versioned_docs/version-2.8.0-deprecated/security-jwt.md index 05e641ed67c17..1fa65b7c27f60 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/security-jwt.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/security-jwt.md @@ -32,11 +32,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -305,7 +305,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/security-kerberos.md b/site2/website/versioned_docs/version-2.8.0-deprecated/security-kerberos.md index 670586a9d1691..c49fa3bea1fce 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/security-kerberos.md @@ -373,7 +373,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/security-overview.md b/site2/website/versioned_docs/version-2.8.0-deprecated/security-overview.md index f8e8eac98f805..227dd5b5d4fc3 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/security-overview.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/security-overview.md @@ -21,7 +21,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/security-tls-authentication.md b/site2/website/versioned_docs/version-2.8.0-deprecated/security-tls-authentication.md index f3c32a05d9751..85d2240f41306 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/security-tls-authentication.md @@ -7,9 +7,9 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -103,7 +103,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -123,7 +123,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/security-tls-keystore.md b/site2/website/versioned_docs/version-2.8.0-deprecated/security-tls-keystore.md index c4474c3c409cf..b09500e3daf04 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/security-tls-keystore.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/security-tls-keystore.md @@ -7,7 +7,7 @@ original_id: security-tls-keystore ## Overview -Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication) between clients and Apache Pulsar service. +Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication.md) between clients and Apache Pulsar service. By default it uses PEM format file configuration. This page tries to describe use [KeyStore](https://en.wikipedia.org/wiki/Java_KeyStore) type configure for TLS. @@ -165,7 +165,7 @@ This is similar to [TLS encryption configuing for client with PEM type](security For a a minimal configuration, user need to provide the TrustStore information. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties @@ -258,7 +258,7 @@ webSocketServiceEnabled=false Besides the TLS encryption configuring. The main work is configuring the KeyStore, which contains a valid CN as client role, for client. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/security-tls-transport.md b/site2/website/versioned_docs/version-2.8.0-deprecated/security-tls-transport.md index 92e8dc3c3d280..2cad17a78c350 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,9 +27,9 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). -`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ## Create TLS certificates @@ -130,7 +130,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Configure broker -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -202,7 +202,7 @@ The examples below show that hostname verification is disabled for the CLI tools ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/security-token-admin.md b/site2/website/versioned_docs/version-2.8.0-deprecated/security-token-admin.md index 1a438dc7923ee..a265f6320d28f 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/security-token-admin.md @@ -163,7 +163,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.8.0-deprecated/sql-deployment-configurations.md index 67f7ab17c77d0..02d8bc78f6cb9 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/sql-deployment-configurations.md @@ -64,7 +64,7 @@ Since Pulsar SQL is powered by [Trino (formerly Presto SQL)](https://trino.io), :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/sql-overview.md b/site2/website/versioned_docs/version-2.8.0-deprecated/sql-overview.md index 4a4d5f01b0bc5..0235d0256c5f1 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/sql-overview.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/sql-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Overview" original_id: sql-overview --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL)](https://trino.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL.md)](https://trino.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/standalone-docker.md b/site2/website/versioned_docs/version-2.8.0-deprecated/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/standalone.md b/site2/website/versioned_docs/version-2.8.0-deprecated/standalone.md index d114c017d7c00..25afa11a91b11 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/standalone.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -65,7 +65,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -74,7 +74,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -123,7 +123,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -137,7 +137,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -170,12 +170,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -208,7 +208,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.8.0-deprecated/tiered-storage-overview.md b/site2/website/versioned_docs/version-2.8.0-deprecated/tiered-storage-overview.md index 7c6528ce2b36c..c635034f463b4 100644 --- a/site2/website/versioned_docs/version-2.8.0-deprecated/tiered-storage-overview.md +++ b/site2/website/versioned_docs/version-2.8.0-deprecated/tiered-storage-overview.md @@ -13,9 +13,9 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws). + - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws.md). - - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs). + - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs.md). ::: @@ -25,7 +25,7 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem). + For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem.md). ::: diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/admin-api-brokers.md b/site2/website/versioned_docs/version-2.8.1-deprecated/admin-api-brokers.md index d51a71e160366..4af4363850efd 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/admin-api-brokers.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/admin-api-brokers.md @@ -18,7 +18,7 @@ Pulsar brokers consist of two components: [Brokers](reference-terminology.md#broker) can be managed via: -* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/brokers` endpoint of the admin {@inject: rest:REST:/} API * The `brokers` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin.html} object in the [Java API](client-libraries-java.md) @@ -166,7 +166,7 @@ One way to configure a Pulsar [broker](reference-terminology.md#broker) is to su But since all broker configuration in Pulsar is stored in ZooKeeper, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values. -* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. +* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. * In the Pulsar admin {@inject: rest:REST:/} API, dynamic configuration is managed through the `/admin/v2/brokers/configuration` endpoint. ### Update dynamic configuration diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/admin-api-clusters.md b/site2/website/versioned_docs/version-2.8.1-deprecated/admin-api-clusters.md index 3eb238506cc69..e0e9fb5f91f65 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/admin-api-clusters.md @@ -16,7 +16,7 @@ servers (aka [bookies](reference-terminology.md#bookie)), and a [ZooKeeper](http Clusters can be managed via: -* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/clusters` endpoint of the admin {@inject: rest:REST:/} API * The `clusters` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md) @@ -79,7 +79,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -103,11 +103,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-cgo.md b/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-cgo.md index c79f7bb965473..f352f942b7714 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-cgo.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-cgo.md @@ -24,7 +24,7 @@ Currently, the following Go clients are maintained in two repositories. ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Install go package @@ -57,7 +57,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -75,7 +75,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -236,14 +236,14 @@ Parameter | Description | Default `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | `Properties`| Attach a set of application defined properties to the producer. This properties will be visible in the topic stats | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | Set the number of max pending messages across all the partitions. This setting will be used to lower the max pending messages for each partition `MaxPendingMessages(int)`, if the total exceeds the configured value.| `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | `Batching` | Control whether automatic batching of messages is enabled for the producer. | false `BatchingMaxPublishDelay` | Set the time period within which the messages sent will be batched (default: 1ms) if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or until | 1ms `BatchingMaxMessages` | Set the maximum number of messages permitted in a batch. (default: 1000) If set to a value greater than 1, messages will be queued until this threshold is reached or batch interval has elapsed | 1000 @@ -512,7 +512,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-cpp.md index 3c94293b7929c..455cf02116d50 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-cpp.md @@ -602,7 +602,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema This section describes some examples about schema. For more information about -schema, see [Pulsar schema](schema-get-started). +schema, see [Pulsar schema](schema-get-started.md). ### Avro schema diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-dotnet.md b/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-dotnet.md index ade664cd95277..b574fa0b2e5ed 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-dotnet.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-dotnet.md @@ -170,7 +170,7 @@ var client = PulsarClient.Builder() Currently, the Pulsar C# client supports the TLS (Transport Layer Security) and JWT (JSON Web Token) authentication. -If you have followed [Authentication using TLS](security-tls-authentication), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: +If you have followed [Authentication using TLS](security-tls-authentication.md), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: 1. Create an unencrypted and password-less pfx file. diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-go.md b/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-go.md index fbb2573aa5db8..6281b03dd8c80 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-go.md @@ -35,7 +35,7 @@ import "github.com/apache/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -61,7 +61,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -844,7 +844,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker @@ -864,7 +864,7 @@ opts := pulsar.ClientOptions{ ## OAuth2 authentication -To use [OAuth2 authentication](security-oauth2), you'll need to configure your client to perform the following operations. +To use [OAuth2 authentication](security-oauth2.md), you'll need to configure your client to perform the following operations. This example shows how to configure OAuth2 authentication. ```go diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-java.md b/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-java.md index e57c83f7963f4..0ff9a22936faf 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-java.md @@ -5,7 +5,7 @@ sidebar_label: "Java" original_id: client-libraries-java --- -You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), and [reader](#reader) of a Java client are thread-safe. @@ -14,10 +14,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Includes both `pulsar-client` and `pulsar-client-admin`

    Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.

    In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -57,7 +57,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -83,7 +83,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -114,7 +114,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -240,7 +240,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -827,7 +827,7 @@ Total hash range size is 65536, so the max end of the range should be less than ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -945,7 +945,7 @@ For example of ProtobufNativeSchema, see [`SchemaDefinition` in `Complex type`]( ## Authentication -Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2). You can use the Pulsar Java client with all of them. +Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2.md). You can use the Pulsar Java client with all of them. ### TLS Authentication @@ -972,7 +972,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` @@ -1009,7 +1009,7 @@ PulsarClient client = PulsarClient.builder() ### Oauth2 -The following example shows how to use [Oauth2](security-oauth2) as an authentication provider for the Pulsar Java client. +The following example shows how to use [Oauth2](security-oauth2.md) as an authentication provider for the Pulsar Java client. You can use the factory method to configure authentication for Pulsar Java client. diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-node.md b/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-node.md index 20ec2432dd38e..e24032946bdcd 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-node.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-node.md @@ -48,7 +48,7 @@ Also, this library works only in Node.js 10.x or later because it uses the [`nod ::: ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here is an example for `localhost`: @@ -66,7 +66,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication), the URL looks like this: +If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication.md), the URL looks like this: ```http @@ -101,7 +101,7 @@ The following configurable parameters are available for Pulsar clients: | Parameter | Description | Default | | :-------- | :---------- | :------ | | `serviceUrl` | The connection URL for the Pulsar cluster. See [above](#connection-urls) for more info. | | -| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication) for more info. | | +| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication.md) for more info. | | | `operationTimeoutSeconds` | The timeout for Node.js client operations (creating producers, subscribing to and unsubscribing from [topics](reference-terminology.md#topic)). Retries occur until this threshold is reached, at which point the operation fails. | 30 | | `ioThreads` | The number of threads to use for handling connections to Pulsar [brokers](reference-terminology.md#broker). | 1 | | `messageListenerThreads` | The number of threads used by message listeners ([consumers](#consumers) and [readers](#readers)). | 1 | @@ -154,7 +154,7 @@ Pulsar Node.js producers have the following methods available: | :-------- | :---------- | :------ | | `topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer publishes messages. The topic format is `` or `//`. For example, `sample/ns1/my-topic`. | | | `producerName` | A name for the producer. If you do not explicitly assign a name, Pulsar automatically generates a globally unique name. If you choose to explicitly assign a name, it needs to be unique across *all* Pulsar clusters, otherwise the creation operation throws an error. | | -| `sendTimeoutMs` | When publishing a message to a topic, the producer waits for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error is thrown. If you set `sendTimeoutMs` to -1, the timeout is set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30000 | +| `sendTimeoutMs` | When publishing a message to a topic, the producer waits for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error is thrown. If you set `sendTimeoutMs` to -1, the timeout is set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30000 | | `initialSequenceId` | The initial sequence ID of the message. When producer send message, add sequence ID to message. The ID is increased each time to send. | | | `maxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `send` method fails *unless* `blockIfQueueFull` is set to `true`. | 1000 | | `maxPendingMessagesAcrossPartitions` | The maximum size of the sum of partition's pending queue. | 50000 | diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-python.md b/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-python.md index 7ebc6bec64e5d..f30cf55387d92 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-websocket.md index bc13b437a97c1..ebdb9bc1cd18f 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/client-libraries-websocket.md @@ -5,14 +5,14 @@ sidebar_label: "WebSocket" original_id: client-libraries-websocket --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-architecture-overview.md index 0c7d44260ddeb..f3e75c3e307e0 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-architecture-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Architecture" original_id: concepts-architecture-overview --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -17,20 +17,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers. The producers connect to the brokers to publish messages and the consumers connect to the brokers to consume the messages. -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -40,9 +40,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -136,17 +136,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -154,7 +154,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-authentication.md b/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-authentication.md index b375ecb16a2d8..f6307890c904a 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-authentication.md @@ -5,5 +5,5 @@ sidebar_label: "Authentication and Authorization" original_id: concepts-authentication --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-clients.md b/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-clients.md index b68f76a2d8b08..4040624f7d636 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-clients.md @@ -5,12 +5,12 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -38,7 +38,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-messaging.md b/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-messaging.md index 92ff067f70374..045d294e96e6e 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-messaging.md @@ -21,8 +21,8 @@ Messages are the basic "unit" of Pulsar. The following table lists the component Component | Description :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer who produces the message. If you do not specify a producer name, the default name is used. Sequence ID | Each Pulsar message belongs to an ordered sequence on its topic. The sequence ID of the message is its order in that sequence. @@ -50,7 +50,7 @@ The default size of a message is 5 MB. You can configure the max size of a messa ``` -> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol). +> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -147,7 +147,7 @@ Messages are received from [brokers](reference-terminology.md#broker) either syn ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -384,7 +384,7 @@ A [subscription](#subscriptions) can have one or more consumers. When a consumer #### When to use -By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry). +By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry.md). #### How to use @@ -488,7 +488,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -500,7 +500,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -538,7 +538,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -625,7 +625,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -648,12 +648,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication.md). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-multi-tenancy.md index 730458c904618..93a59557b2efc 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-multi-tenancy.md @@ -5,7 +5,7 @@ sidebar_label: "Multi Tenancy" original_id: concepts-multi-tenancy --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -21,7 +21,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -29,7 +29,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-overview.md b/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-overview.md index c1262b38daf16..e8a2f4b9d321a 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-replication.md b/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-replication.md index 6e23962e99771..799f0eb4d92c6 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-replication.md @@ -5,5 +5,5 @@ sidebar_label: "Geo Replication" original_id: concepts-replication --- -Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that. +Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo.md) in Pulsar enables you to do that. diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-tiered-storage.md index 0b45b0a2ed501..f6988e53a8cd4 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-topic-compaction.md index e402c965384fd..34b7ed7fbbd31 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/concepts-topic-compaction.md @@ -7,7 +7,7 @@ original_id: concepts-topic-compaction Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -24,7 +24,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.8.1-deprecated/cookbooks-compaction.md index 6fcbc197f9ecb..dfa314727241a 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.8.1-deprecated/cookbooks-non-persistent.md index 391569a8cbc6b..178301e86eb8d 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/cookbooks-non-persistent.md @@ -41,7 +41,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -55,7 +55,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.8.1-deprecated/cookbooks-partitioned.md index 7882fb9025312..fb9ac354cc6d6 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/cookbooks-partitioned.md @@ -4,4 +4,4 @@ title: Partitioned topics sidebar_label: "Partitioned Topics" original_id: cookbooks-partitioned --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.8.1-deprecated/cookbooks-retention-expiry.md index af4be6bb2d9c6..192fbe6c53277 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/cookbooks-retention-expiry.md @@ -22,7 +22,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-aws.md b/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-aws.md index 071effeffde24..93c389b56e2cf 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-aws.md @@ -5,7 +5,7 @@ sidebar_label: "Amazon Web Services" original_id: deploy-aws --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -210,7 +210,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors) , edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md) , edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-bare-metal-multi-cluster.md index dacd571365aa2..1b23eea07a20b 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-bare-metal-multi-cluster.md @@ -9,17 +9,17 @@ original_id: deploy-bare-metal-multi-cluster 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using it in a startup or on a single team, you had better opt for a single cluster. For instructions on deploying a single cluster, -see the guide [here](deploy-bare-metal). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo). Deploying a multi-cluster Pulsar instance involves the following basic steps: +A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md). Deploying a multi-cluster Pulsar instance involves the following basic steps: * Deploying two separate [ZooKeeper](#deploy-zookeeper) quorums: a [local](#deploy-local-zookeeper) quorum for each cluster in the instance and a [configuration store](#configuration-store) quorum for instance-wide tasks * Initializing [cluster metadata](#cluster-metadata-initialization) for each cluster @@ -29,7 +29,7 @@ A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You If you want to deploy a single Pulsar cluster, see [Clusters and Brokers](getting-started-standalone.md#start-the-cluster). > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). ## System requirement @@ -74,7 +74,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -252,7 +252,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -371,17 +371,17 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). -You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > #### Service discovery already provided by many scheduling systems -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ### Service discovery setup -The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol) of Pulsar. +The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol.md) of Pulsar. To get started setting up the built-in service of discovery of Pulsar, you need to change a few parameters in the [`conf/discovery.conf`](reference-configuration.md#service-discovery) configuration file. Set the [`zookeeperServers`](reference-configuration.md#service-discovery-zookeeperServers) parameter to the ZooKeeper quorum connection string of the cluster and the [`configurationStoreServers`](reference-configuration.md#service-discovery-configurationStoreServers) setting to the [configuration store](reference-terminology.md#configuration-store) quorum connection string. @@ -406,7 +406,7 @@ $ bin/pulsar-daemon start discovery ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-bare-metal.md index f856f9f8b1ca6..bdd05f24f2566 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using Pulsar in a startup or on a single team, it is simplest to opt for a single cluster. If you do need to run a multi-cluster Pulsar instance, -see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -107,7 +107,7 @@ The extracted directory contains the following subdirectories: Directory | Contains :---------|:-------- -`bin` |[command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` |[command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `data` | The data storage directory that ZooKeeper and BookKeeper use `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -261,9 +261,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (you had better not use a different port). -`--web-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). +`--web-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should not use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (you had better not use a different port). -`--broker-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). +`--broker-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). > If you do not have a DNS server, you can use multi-host format in the service URL with the following settings: @@ -403,7 +403,7 @@ webServicePortTls=8443 ### Enable Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -421,7 +421,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker). +If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker.md). ### Start Brokers diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-dcos.md b/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-dcos.md index f5f8d1faa7dbe..952d5f47e30fa 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -If you want to enable all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of +If you want to enable all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-docker.md b/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-docker.md index 64a0939aa787b..8348d78deb237 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-docker.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-docker.md @@ -57,4 +57,4 @@ docker network connect pulsar broker To check whether the containers are successfully connected to the network, enter the `docker network inspect pulsar` command. -For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal). +For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal.md). diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-kubernetes.md index dc7123d000467..1aefc6ad79f71 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-kubernetes.md @@ -6,6 +6,6 @@ original_id: deploy-kubernetes --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-monitoring.md b/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-monitoring.md index ae2ac29943331..f9fe0e0bb97be 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/deploy-monitoring.md @@ -113,7 +113,7 @@ http://$FUNCTIONS_WORKER_ADDRESS:$WORKER_PORT/metrics: You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy). +When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy.md). ## Dashboards @@ -121,7 +121,7 @@ When you collect time series statistics, the major problem is to make sure the n ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager). +The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/develop-schema.md b/site2/website/versioned_docs/version-2.8.1-deprecated/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/develop-schema.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/functions-deploy.md b/site2/website/versioned_docs/version-2.8.1-deprecated/functions-deploy.md index 10a8339edb8d5..9c47208f68fa0 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/functions-deploy.md @@ -9,12 +9,12 @@ original_id: functions-deploy To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface @@ -179,7 +179,7 @@ $ bin/pulsar-admin functions create \ Package management enables version management and simplifies the upgrade and rollback processes for Functions, Sinks, and Sources. When you use the same function, sink and source in different namespaces, you can upload them to a common package management system. -To use [Package management service](admin-api-packages), ensure that the package management service has been enabled in your cluster by setting the following properties in `broker.conf`. +To use [Package management service](admin-api-packages.md), ensure that the package management service has been enabled in your cluster by setting the following properties in `broker.conf`. > Note: Package management service is not enabled by default. diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/functions-develop.md b/site2/website/versioned_docs/version-2.8.1-deprecated/functions-develop.md index 36ea4b4a21073..2e29aa1c47400 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/functions-develop.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/functions-develop.md @@ -1144,7 +1144,7 @@ You can monitor Pulsar Functions that have been deployed with the following meth - Check the metrics provided by Pulsar. - Pulsar Functions expose the metrics that can be collected and used for monitoring the health of **Java, Python, and Go** functions. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar Functions expose the metrics that can be collected and used for monitoring the health of **Java, Python, and Go** functions. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. For the complete list of the function metrics, see [here](reference-metrics.md#pulsar-functions). @@ -1214,7 +1214,7 @@ Currently, the feature is not available in Go. ## Security -If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker). For more details, refer to [Security settings](functions-worker.md#security-settings). +If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings). Pulsar Functions can support the following providers: diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/functions-overview.md b/site2/website/versioned_docs/version-2.8.1-deprecated/functions-overview.md index 25dc6029acf70..816d301e0fd0e 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/functions-overview.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/functions-overview.md @@ -164,7 +164,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/functions-package.md b/site2/website/versioned_docs/version-2.8.1-deprecated/functions-package.md index 82346aa8c5098..a995d5c158877 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/functions-package.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/functions-package.md @@ -15,7 +15,7 @@ Currently, the window function is not available in Python and Go. ## Prerequisite -Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm). +Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm.md). To check whether the Docker image starts, you can use the `docker ps` command. diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/functions-worker.md b/site2/website/versioned_docs/version-2.8.1-deprecated/functions-worker.md index 78f4f19487951..49fc76b30bdaa 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/functions-worker.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/functions-worker.md @@ -4,7 +4,7 @@ title: Deploy and manage functions worker sidebar_label: "Setup: Pulsar Functions Worker" original_id: functions-worker --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -179,7 +179,7 @@ brokerClientTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -199,7 +199,7 @@ authenticationProviders: [ provider1, provider2 ] ``` For *TLS Authentication* provider, follow the example below to add the necessary settings. -See [TLS Authentication](security-tls-authentication) for more details. +See [TLS Authentication](security-tls-authentication.md) for more details. ``` @@ -223,7 +223,7 @@ properties: ``` For *Token Authentication* provider, add necessary settings for `properties` if needed. -See [Token Authentication](security-jwt) for more details. +See [Token Authentication](security-jwt.md) for more details. Note: key files must be DER-encoded ``` @@ -262,7 +262,7 @@ superUserRoles: You can use the public and private key pair that the application configures to perform encryption. Only the consumers with a valid key can decrypt the encrypted messages. -To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption). +To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption.md). We include the relevant configuration information of `CryptoConfig` into `ProducerConfig`. The specific configurable field information about `CryptoConfig` is as follows: diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/getting-started-docker.md b/site2/website/versioned_docs/version-2.8.1-deprecated/getting-started-docker.md index b9f12f0b52688..4f20971d75330 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/getting-started-docker.md @@ -50,8 +50,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -110,7 +110,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/getting-started-helm.md b/site2/website/versioned_docs/version-2.8.1-deprecated/getting-started-helm.md index 65a9fc04fe568..440087c275c05 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/getting-started-helm.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/getting-started-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/getting-started-pulsar.md b/site2/website/versioned_docs/version-2.8.1-deprecated/getting-started-pulsar.md index 11c5e66cb62c2..752590f57b558 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/getting-started-pulsar.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/getting-started-pulsar.md @@ -5,13 +5,13 @@ sidebar_label: "Pulsar 2.0" original_id: pulsar-2.0 --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/getting-started-standalone.md b/site2/website/versioned_docs/version-2.8.1-deprecated/getting-started-standalone.md index 0e40286d71dcb..cea47efd08d4b 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -65,7 +65,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -74,7 +74,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -123,7 +123,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -137,7 +137,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -170,12 +170,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -208,7 +208,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/helm-install.md b/site2/website/versioned_docs/version-2.8.1-deprecated/helm-install.md index edb12260f010e..1f4d5eb69d5dd 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/helm-install.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/helm-install.md @@ -21,7 +21,7 @@ Before deploying Pulsar, you need to prepare your environment. ### Tools -Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools) on your computer. +Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools.md) on your computer. ## Cloud cluster preparation @@ -37,8 +37,8 @@ To create and connect to the Kubernetes cluster, follow the instructions: ## Pulsar deployment -Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy). +Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy.md). ## Pulsar upgrade -To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade). +To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade.md). diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/helm-overview.md b/site2/website/versioned_docs/version-2.8.1-deprecated/helm-overview.md index 9cde85e2d18d5..385d535e319b6 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/helm-overview.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/helm-overview.md @@ -52,9 +52,9 @@ It includes support for: ## Pulsar Helm chart quick start -To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install). +This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install.md). ## Troubleshooting @@ -64,7 +64,7 @@ We have done our best to make these charts as seamless as possible. Occasionally The Apache Pulsar Helm chart contains all required dependencies. -If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm) for your first iteration. +If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm.md) for your first iteration. 1. [Preparation](helm-prepare.md) 2. [Deployment](helm-deploy.md) @@ -82,7 +82,7 @@ helm upgrade apache/pulsar -f pulsar.yaml ``` -For more detailed information, see [Upgrading](helm-upgrade). +For more detailed information, see [Upgrading](helm-upgrade.md). ## Uninstallation diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/helm-prepare.md b/site2/website/versioned_docs/version-2.8.1-deprecated/helm-prepare.md index 95d7e668f9eca..5e9f2f9ef4f68 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/helm-prepare.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/helm-prepare.md @@ -89,4 +89,4 @@ PROJECT= USE_LOCAL_SSD=true LOCAL_SSD_COUNT= ## Next Steps -Continue with the [installation of the chart](helm-deploy) once you have the cluster up and running. +Continue with the [installation of the chart](helm-deploy.md) once you have the cluster up and running. diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/helm-tools.md b/site2/website/versioned_docs/version-2.8.1-deprecated/helm-tools.md index efd61eae32506..6ba89006913b6 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/helm-tools.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/helm-tools.md @@ -25,7 +25,7 @@ You can get Helm from the project's [releases page](https://github.com/helm/helm ### Next steps -Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare). +Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare.md). ## Additional information diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/helm-upgrade.md b/site2/website/versioned_docs/version-2.8.1-deprecated/helm-upgrade.md index e39cf97407355..7d671e6bfb3c1 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/helm-upgrade.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/helm-upgrade.md @@ -20,7 +20,7 @@ You can retrieve your previous `--set` arguments cleanly, with `helm get values To upgrade Apache Pulsar to a newer version, follow these steps: 1. Check the change log for the specific version you would like to upgrade to. -2. Go through [deployment documentation](helm-deploy) step by step. +2. Go through [deployment documentation](helm-deploy.md) step by step. 3. Extract your previous `--set` arguments with the following command. ```bash diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/io-cdc.md b/site2/website/versioned_docs/version-2.8.1-deprecated/io-cdc.md index 20f16ae0d6904..e6e662884826d 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/io-cdc.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/io-cdc.md @@ -13,8 +13,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/io-develop.md b/site2/website/versioned_docs/version-2.8.1-deprecated/io-develop.md index a20206fbf6a06..d6f4f8261ac82 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/io-develop.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/io-develop.md @@ -14,15 +14,15 @@ import TabItem from '@theme/TabItem'; This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -275,7 +275,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -304,7 +304,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). @@ -379,7 +379,7 @@ Pulsar connectors enable you to move data in and out of Pulsar easily. It is imp - Check the metrics provided by Pulsar. - Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. - Set and check your customized metrics. diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/io-overview.md b/site2/website/versioned_docs/version-2.8.1-deprecated/io-overview.md index 0070a015296db..3db5ee34042d3 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/io-overview.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/io-overview.md @@ -158,7 +158,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/io-quickstart.md b/site2/website/versioned_docs/version-2.8.1-deprecated/io-quickstart.md index b8fea1560bdeb..8474c93f51336 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you are able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you are able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However, all the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -109,7 +109,7 @@ This section demonstrates how to connect Pulsar to Cassandra. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink.md). ::: @@ -236,11 +236,11 @@ You can create a configuration file through one of the following methods. ``` -For more information, see [Cassandra sink connector](io-cassandra-sink). +For more information, see [Cassandra sink connector](io-cassandra-sink.md). ### Create a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on them. Run the following command to create a Cassandra sink connector with sink type _cassandra_ and the config file _examples/cassandra-sink.yml_ created previously. @@ -267,7 +267,7 @@ as a Pulsar Function and writes the messages produced in the topic _test_cassand ### Inspect a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * Get the information of a Cassandra sink. @@ -431,7 +431,7 @@ to monitor a connector and perform other operations on it. ### Delete a Cassandra Sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. ```bash @@ -454,7 +454,7 @@ This section demonstrates how to connect Pulsar to PostgreSQL. ::: ->For more information, see [JDBC sink connector](io-jdbc-sink). +>For more information, see [JDBC sink connector](io-jdbc-sink.md). ### Setup a PostgreSQL cluster @@ -614,7 +614,7 @@ In this section, you need to configure a JDBC sink connector. ### Create a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on it. This example creates a sink connector and specifies the desired information. @@ -660,7 +660,7 @@ The sink has been created successfully if the following message appears. ### Inspect a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * List all running JDBC sink(s). @@ -780,7 +780,7 @@ to monitor a connector and perform other operations on it. ### Stop a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to stop a connector and perform other operations on it. ```bash @@ -808,7 +808,7 @@ The sink instance has been stopped successfully if the following message disappe ### Restart a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to restart a connector and perform other operations on it. ```bash @@ -844,7 +844,7 @@ Note that `pulsar-admin sinks localrun options` **runs a sink connector locally* ### Update a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to update a connector and perform other operations on it. This example updates the parallelism of the _pulsar-postgres-jdbc-sink_ sink connector to 2. @@ -912,7 +912,7 @@ The result shows that the parallelism is 2. ### Delete a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. This example deletes the _pulsar-postgres-jdbc-sink_ sink connector. diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/io-use.md b/site2/website/versioned_docs/version-2.8.1-deprecated/io-use.md index b6409d4420ad7..da9ed746c4d37 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/io-use.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/io-use.md @@ -15,7 +15,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/kubernetes-helm.md b/site2/website/versioned_docs/version-2.8.1-deprecated/kubernetes-helm.md index a9f768e6a96b3..ea92a0968cd7d 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/kubernetes-helm.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/kubernetes-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/reference-cli-tools.md b/site2/website/versioned_docs/version-2.8.1-deprecated/reference-cli-tools.md index ecd6363453e3c..309009f635e29 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/reference-configuration.md b/site2/website/versioned_docs/version-2.8.1-deprecated/reference-configuration.md index 3cea1d7aaffff..68124dd94f29e 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/reference-configuration.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/reference-configuration.md @@ -8,7 +8,7 @@ original_id: reference-configuration -You can manage Pulsar configuration by configuration files in the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a Pulsar [installation](getting-started-standalone). +You can manage Pulsar configuration by configuration files in the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a Pulsar [installation](getting-started-standalone.md). - [BookKeeper](#bookkeeper) - [Broker](#broker) diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/schema-get-started.md b/site2/website/versioned_docs/version-2.8.1-deprecated/schema-get-started.md index 97e270586a26d..afacb0fa51f2e 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/schema-get-started.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/schema-get-started.md @@ -17,7 +17,7 @@ Applications typically adopt one of the following approaches to guarantee type s #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ### Client-side approach diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/schema-manage.md b/site2/website/versioned_docs/version-2.8.1-deprecated/schema-manage.md index 4ae363882b2d5..c588aae619eee 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/schema-manage.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/schema-manage.md @@ -160,7 +160,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/schema-understand.md b/site2/website/versioned_docs/version-2.8.1-deprecated/schema-understand.md index 153d96e6e71cb..a86b02add435e 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/schema-understand.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/schema-understand.md @@ -459,7 +459,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```java diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/security-athenz.md b/site2/website/versioned_docs/version-2.8.1-deprecated/security-athenz.md index 947c3f470be46..8a39fe25316d0 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/security-athenz.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/security-athenz.md @@ -76,7 +76,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/security-authorization.md b/site2/website/versioned_docs/version-2.8.1-deprecated/security-authorization.md index cc9c03d5fc1dd..9cfd7c8c203f6 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/security-authorization.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/security-authorization.md @@ -27,7 +27,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -59,7 +59,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -87,7 +87,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication @@ -117,7 +117,7 @@ PulsarAdmin admin = PulsarAdmin.builder() When a client is identified with multiple roles in a token (the type of role claim in the token is an array) during the authentication process, Pulsar supports to check the permissions of all the roles and further authorize the client as long as one of its roles has the required permissions. > **Note**
    -> This authorization method is only compatible with [JWT authentication](security-jwt). +> This authorization method is only compatible with [JWT authentication](security-jwt.md). To enable this authorization method, configure the authorization provider as `MultiRolesTokenAuthorizationProvider` in the `conf/broker.conf` file. diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/security-bouncy-castle.md b/site2/website/versioned_docs/version-2.8.1-deprecated/security-bouncy-castle.md index abc25c71b8ae2..be937055d8e31 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/security-bouncy-castle.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/security-bouncy-castle.md @@ -16,7 +16,7 @@ In Pulsar, security and crypto have dependencies on BouncyCastle Jars. For the d `Bouncy Castle` provides both [FIPS](https://www.bouncycastle.org/fips_faq.html) and non-FIPS version. But in a JVM, you can not include both of the 2 versions, and you need to exclude the current version before include the other. -In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. +In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption.md). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. ## How BouncyCastle modules packaged in Pulsar diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/security-extending.md b/site2/website/versioned_docs/version-2.8.1-deprecated/security-extending.md index c088e3a2c1c2c..e7484453b8beb 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/security-extending.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/security-extending.md @@ -9,7 +9,7 @@ Pulsar provides a way to use custom authentication and authorization mechanisms. ## Authentication -Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview). +Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview.md). You can use a custom authentication mechanism by providing the implementation in the form of two plugins. One plugin is for the Client library and the other plugin is for the Pulsar Proxy and/or Pulsar Broker to validate the credentials. diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/security-jwt.md b/site2/website/versioned_docs/version-2.8.1-deprecated/security-jwt.md index 05e641ed67c17..1fa65b7c27f60 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/security-jwt.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/security-jwt.md @@ -32,11 +32,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -305,7 +305,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/security-kerberos.md b/site2/website/versioned_docs/version-2.8.1-deprecated/security-kerberos.md index 670586a9d1691..c49fa3bea1fce 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/security-kerberos.md @@ -373,7 +373,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/security-overview.md b/site2/website/versioned_docs/version-2.8.1-deprecated/security-overview.md index f8e8eac98f805..227dd5b5d4fc3 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/security-overview.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/security-overview.md @@ -21,7 +21,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/security-tls-authentication.md b/site2/website/versioned_docs/version-2.8.1-deprecated/security-tls-authentication.md index f3c32a05d9751..85d2240f41306 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/security-tls-authentication.md @@ -7,9 +7,9 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -103,7 +103,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -123,7 +123,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/security-tls-keystore.md b/site2/website/versioned_docs/version-2.8.1-deprecated/security-tls-keystore.md index c4474c3c409cf..b09500e3daf04 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/security-tls-keystore.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/security-tls-keystore.md @@ -7,7 +7,7 @@ original_id: security-tls-keystore ## Overview -Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication) between clients and Apache Pulsar service. +Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication.md) between clients and Apache Pulsar service. By default it uses PEM format file configuration. This page tries to describe use [KeyStore](https://en.wikipedia.org/wiki/Java_KeyStore) type configure for TLS. @@ -165,7 +165,7 @@ This is similar to [TLS encryption configuing for client with PEM type](security For a a minimal configuration, user need to provide the TrustStore information. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties @@ -258,7 +258,7 @@ webSocketServiceEnabled=false Besides the TLS encryption configuring. The main work is configuring the KeyStore, which contains a valid CN as client role, for client. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/security-tls-transport.md b/site2/website/versioned_docs/version-2.8.1-deprecated/security-tls-transport.md index 92e8dc3c3d280..2cad17a78c350 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,9 +27,9 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). -`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ## Create TLS certificates @@ -130,7 +130,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Configure broker -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -202,7 +202,7 @@ The examples below show that hostname verification is disabled for the CLI tools ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/security-token-admin.md b/site2/website/versioned_docs/version-2.8.1-deprecated/security-token-admin.md index 1a438dc7923ee..a265f6320d28f 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/security-token-admin.md @@ -163,7 +163,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.8.1-deprecated/sql-deployment-configurations.md index 67f7ab17c77d0..02d8bc78f6cb9 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/sql-deployment-configurations.md @@ -64,7 +64,7 @@ Since Pulsar SQL is powered by [Trino (formerly Presto SQL)](https://trino.io), :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/sql-overview.md b/site2/website/versioned_docs/version-2.8.1-deprecated/sql-overview.md index 4a4d5f01b0bc5..0235d0256c5f1 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/sql-overview.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/sql-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Overview" original_id: sql-overview --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL)](https://trino.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL.md)](https://trino.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/standalone-docker.md b/site2/website/versioned_docs/version-2.8.1-deprecated/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/standalone.md b/site2/website/versioned_docs/version-2.8.1-deprecated/standalone.md index d114c017d7c00..25afa11a91b11 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/standalone.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -65,7 +65,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -74,7 +74,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -123,7 +123,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -137,7 +137,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -170,12 +170,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -208,7 +208,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.8.1-deprecated/tiered-storage-overview.md b/site2/website/versioned_docs/version-2.8.1-deprecated/tiered-storage-overview.md index 7c6528ce2b36c..c635034f463b4 100644 --- a/site2/website/versioned_docs/version-2.8.1-deprecated/tiered-storage-overview.md +++ b/site2/website/versioned_docs/version-2.8.1-deprecated/tiered-storage-overview.md @@ -13,9 +13,9 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws). + - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws.md). - - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs). + - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs.md). ::: @@ -25,7 +25,7 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem). + For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem.md). ::: diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/admin-api-brokers.md b/site2/website/versioned_docs/version-2.8.2-deprecated/admin-api-brokers.md index 3effc4a192326..aea8f70f87a7d 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/admin-api-brokers.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/admin-api-brokers.md @@ -28,7 +28,7 @@ Pulsar brokers consist of two components: [Brokers](reference-terminology.md#broker) can be managed via: -* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/brokers` endpoint of the admin {@inject: rest:REST:/} API * The `brokers` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin.html} object in the [Java API](client-libraries-java.md) @@ -176,7 +176,7 @@ One way to configure a Pulsar [broker](reference-terminology.md#broker) is to su But since all broker configuration in Pulsar is stored in ZooKeeper, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values. -* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. +* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. * In the Pulsar admin {@inject: rest:REST:/} API, dynamic configuration is managed through the `/admin/v2/brokers/configuration` endpoint. ### Update dynamic configuration diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/admin-api-clusters.md b/site2/website/versioned_docs/version-2.8.2-deprecated/admin-api-clusters.md index c4b404fc8e680..ce085345640a4 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/admin-api-clusters.md @@ -26,7 +26,7 @@ servers (aka [bookies](reference-terminology.md#bookie)), and a [ZooKeeper](http Clusters can be managed via: -* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/clusters` endpoint of the admin {@inject: rest:REST:/} API * The `clusters` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md) @@ -89,7 +89,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -113,11 +113,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block > - For the latest and complete information about `Java admin API`, including classes, methods, descriptions, and more, see [Java admin API doc](https://pulsar.apache.org/api/admin/). -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -118,7 +118,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-cgo.md b/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-cgo.md index c79f7bb965473..f352f942b7714 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-cgo.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-cgo.md @@ -24,7 +24,7 @@ Currently, the following Go clients are maintained in two repositories. ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Install go package @@ -57,7 +57,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -75,7 +75,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -236,14 +236,14 @@ Parameter | Description | Default `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | `Properties`| Attach a set of application defined properties to the producer. This properties will be visible in the topic stats | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | Set the number of max pending messages across all the partitions. This setting will be used to lower the max pending messages for each partition `MaxPendingMessages(int)`, if the total exceeds the configured value.| `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | `Batching` | Control whether automatic batching of messages is enabled for the producer. | false `BatchingMaxPublishDelay` | Set the time period within which the messages sent will be batched (default: 1ms) if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or until | 1ms `BatchingMaxMessages` | Set the maximum number of messages permitted in a batch. (default: 1000) If set to a value greater than 1, messages will be queued until this threshold is reached or batch interval has elapsed | 1000 @@ -512,7 +512,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-cpp.md index deebe13bada6d..08622b9e83071 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-cpp.md @@ -372,7 +372,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema -This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started). +This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started.md). ### Create producer with Avro schema diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-dotnet.md b/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-dotnet.md index 4e0afe3d2ff6b..fbec5e473be69 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-dotnet.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-dotnet.md @@ -170,7 +170,7 @@ var client = PulsarClient.Builder() Currently, the Pulsar C# client supports the TLS (Transport Layer Security) and JWT (JSON Web Token) authentication. -If you have followed [Authentication using TLS](security-tls-authentication), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: +If you have followed [Authentication using TLS](security-tls-authentication.md), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: 1. Create an unencrypted and password-less pfx file. diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-go.md b/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-go.md index fbb2573aa5db8..6281b03dd8c80 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-go.md @@ -35,7 +35,7 @@ import "github.com/apache/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -61,7 +61,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -844,7 +844,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker @@ -864,7 +864,7 @@ opts := pulsar.ClientOptions{ ## OAuth2 authentication -To use [OAuth2 authentication](security-oauth2), you'll need to configure your client to perform the following operations. +To use [OAuth2 authentication](security-oauth2.md), you'll need to configure your client to perform the following operations. This example shows how to configure OAuth2 authentication. ```go diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-java.md b/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-java.md index e57c83f7963f4..0ff9a22936faf 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-java.md @@ -5,7 +5,7 @@ sidebar_label: "Java" original_id: client-libraries-java --- -You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), and [reader](#reader) of a Java client are thread-safe. @@ -14,10 +14,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Includes both `pulsar-client` and `pulsar-client-admin`

    Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.

    In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -57,7 +57,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -83,7 +83,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -114,7 +114,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -240,7 +240,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -827,7 +827,7 @@ Total hash range size is 65536, so the max end of the range should be less than ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -945,7 +945,7 @@ For example of ProtobufNativeSchema, see [`SchemaDefinition` in `Complex type`]( ## Authentication -Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2). You can use the Pulsar Java client with all of them. +Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2.md). You can use the Pulsar Java client with all of them. ### TLS Authentication @@ -972,7 +972,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` @@ -1009,7 +1009,7 @@ PulsarClient client = PulsarClient.builder() ### Oauth2 -The following example shows how to use [Oauth2](security-oauth2) as an authentication provider for the Pulsar Java client. +The following example shows how to use [Oauth2](security-oauth2.md) as an authentication provider for the Pulsar Java client. You can use the factory method to configure authentication for Pulsar Java client. diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-node.md b/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-node.md index 3cb743bc300c5..8031d287c6f2a 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-node.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-node.md @@ -48,7 +48,7 @@ Also, this library works only in Node.js 10.x or later because it uses the [`nod ::: ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here is an example for `localhost`: @@ -66,7 +66,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication), the URL looks like this: +If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication.md), the URL looks like this: ```http @@ -101,7 +101,7 @@ The following configurable parameters are available for Pulsar clients: | Parameter | Description | Default | | :-------- | :---------- | :------ | | `serviceUrl` | The connection URL for the Pulsar cluster. See [above](#connection-urls) for more info. | | -| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication) for more info. | | +| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication.md) for more info. | | | `operationTimeoutSeconds` | The timeout for Node.js client operations (creating producers, subscribing to and unsubscribing from [topics](reference-terminology.md#topic)). Retries occur until this threshold is reached, at which point the operation fails. | 30 | | `ioThreads` | The number of threads to use for handling connections to Pulsar [brokers](reference-terminology.md#broker). | 1 | | `messageListenerThreads` | The number of threads used by message listeners ([consumers](#consumers) and [readers](#readers)). | 1 | @@ -154,7 +154,7 @@ Pulsar Node.js producers have the following methods available: | :-------- | :---------- | :------ | | `topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer publishes messages. The topic format is `` or `//`. For example, `sample/ns1/my-topic`. | | | `producerName` | A name for the producer. If you do not explicitly assign a name, Pulsar automatically generates a globally unique name. If you choose to explicitly assign a name, it needs to be unique across *all* Pulsar clusters, otherwise the creation operation throws an error. | | -| `sendTimeoutMs` | When publishing a message to a topic, the producer waits for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error is thrown. If you set `sendTimeoutMs` to -1, the timeout is set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30000 | +| `sendTimeoutMs` | When publishing a message to a topic, the producer waits for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error is thrown. If you set `sendTimeoutMs` to -1, the timeout is set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30000 | | `initialSequenceId` | The initial sequence ID of the message. When producer send message, add sequence ID to message. The ID is increased each time to send. | | | `maxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `send` method fails *unless* `blockIfQueueFull` is set to `true`. | 1000 | | `maxPendingMessagesAcrossPartitions` | The maximum size of the sum of partition's pending queue. | 50000 | diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-python.md b/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-python.md index 7ebc6bec64e5d..f30cf55387d92 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-websocket.md index 913b6cf5c82a9..e2da41f0461b4 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/client-libraries-websocket.md @@ -5,14 +5,14 @@ sidebar_label: "WebSocket" original_id: client-libraries-websocket --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-architecture-overview.md index 0c7d44260ddeb..f3e75c3e307e0 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-architecture-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Architecture" original_id: concepts-architecture-overview --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -17,20 +17,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers. The producers connect to the brokers to publish messages and the consumers connect to the brokers to consume the messages. -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -40,9 +40,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -136,17 +136,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -154,7 +154,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-authentication.md b/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-authentication.md index b375ecb16a2d8..f6307890c904a 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-authentication.md @@ -5,5 +5,5 @@ sidebar_label: "Authentication and Authorization" original_id: concepts-authentication --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-clients.md b/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-clients.md index b68f76a2d8b08..4040624f7d636 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-clients.md @@ -5,12 +5,12 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -38,7 +38,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-messaging.md b/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-messaging.md index 4be25256b4b36..b76728f109b5d 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-messaging.md @@ -21,8 +21,8 @@ Messages are the basic "unit" of Pulsar. The following table lists the component Component | Description :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer who produces the message. If you do not specify a producer name, the default name is used. Sequence ID | Each Pulsar message belongs to an ordered sequence on its topic. The sequence ID of the message is its order in that sequence. @@ -50,7 +50,7 @@ The default size of a message is 5 MB. You can configure the max size of a messa ``` -> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol). +> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -147,7 +147,7 @@ Messages are received from [brokers](reference-terminology.md#broker) either syn ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -384,7 +384,7 @@ A [subscription](#subscriptions) can have one or more consumers. When a consumer #### When to use -By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry). +By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry.md). #### How to use @@ -488,7 +488,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -500,7 +500,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -538,7 +538,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -625,7 +625,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -648,12 +648,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication.md). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-multi-tenancy.md index 730458c904618..93a59557b2efc 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-multi-tenancy.md @@ -5,7 +5,7 @@ sidebar_label: "Multi Tenancy" original_id: concepts-multi-tenancy --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -21,7 +21,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -29,7 +29,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-overview.md b/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-overview.md index c1262b38daf16..e8a2f4b9d321a 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-replication.md b/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-replication.md index 6e23962e99771..799f0eb4d92c6 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-replication.md @@ -5,5 +5,5 @@ sidebar_label: "Geo Replication" original_id: concepts-replication --- -Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that. +Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo.md) in Pulsar enables you to do that. diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-tiered-storage.md index 0b45b0a2ed501..f6988e53a8cd4 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-topic-compaction.md index e402c965384fd..34b7ed7fbbd31 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/concepts-topic-compaction.md @@ -7,7 +7,7 @@ original_id: concepts-topic-compaction Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -24,7 +24,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.8.2-deprecated/cookbooks-compaction.md index 6fcbc197f9ecb..dfa314727241a 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.8.2-deprecated/cookbooks-non-persistent.md index 391569a8cbc6b..178301e86eb8d 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/cookbooks-non-persistent.md @@ -41,7 +41,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -55,7 +55,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.8.2-deprecated/cookbooks-partitioned.md index 7882fb9025312..fb9ac354cc6d6 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/cookbooks-partitioned.md @@ -4,4 +4,4 @@ title: Partitioned topics sidebar_label: "Partitioned Topics" original_id: cookbooks-partitioned --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.8.2-deprecated/cookbooks-retention-expiry.md index af4be6bb2d9c6..192fbe6c53277 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/cookbooks-retention-expiry.md @@ -22,7 +22,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-aws.md b/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-aws.md index 1c8f1a08d7249..662e99ceb9b09 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-aws.md @@ -5,7 +5,7 @@ sidebar_label: "Amazon Web Services" original_id: deploy-aws --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -210,7 +210,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-bare-metal-multi-cluster.md index dacd571365aa2..1b23eea07a20b 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-bare-metal-multi-cluster.md @@ -9,17 +9,17 @@ original_id: deploy-bare-metal-multi-cluster 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using it in a startup or on a single team, you had better opt for a single cluster. For instructions on deploying a single cluster, -see the guide [here](deploy-bare-metal). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo). Deploying a multi-cluster Pulsar instance involves the following basic steps: +A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md). Deploying a multi-cluster Pulsar instance involves the following basic steps: * Deploying two separate [ZooKeeper](#deploy-zookeeper) quorums: a [local](#deploy-local-zookeeper) quorum for each cluster in the instance and a [configuration store](#configuration-store) quorum for instance-wide tasks * Initializing [cluster metadata](#cluster-metadata-initialization) for each cluster @@ -29,7 +29,7 @@ A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You If you want to deploy a single Pulsar cluster, see [Clusters and Brokers](getting-started-standalone.md#start-the-cluster). > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). ## System requirement @@ -74,7 +74,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -252,7 +252,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -371,17 +371,17 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). -You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > #### Service discovery already provided by many scheduling systems -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ### Service discovery setup -The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol) of Pulsar. +The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol.md) of Pulsar. To get started setting up the built-in service of discovery of Pulsar, you need to change a few parameters in the [`conf/discovery.conf`](reference-configuration.md#service-discovery) configuration file. Set the [`zookeeperServers`](reference-configuration.md#service-discovery-zookeeperServers) parameter to the ZooKeeper quorum connection string of the cluster and the [`configurationStoreServers`](reference-configuration.md#service-discovery-configurationStoreServers) setting to the [configuration store](reference-terminology.md#configuration-store) quorum connection string. @@ -406,7 +406,7 @@ $ bin/pulsar-daemon start discovery ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-bare-metal.md index a7cbd126869dd..8152694e39ee7 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using Pulsar in a startup or on a single team, it is simplest to opt for a single cluster. If you do need to run a multi-cluster Pulsar instance, -see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -107,7 +107,7 @@ The extracted directory contains the following subdirectories: Directory | Contains :---------|:-------- -`bin` |[command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` |[command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `data` | The data storage directory that ZooKeeper and BookKeeper use `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -261,9 +261,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (you had better not use a different port). -`--web-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). +`--web-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should not use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (you had better not use a different port). -`--broker-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). +`--broker-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). > If you do not have a DNS server, you can use multi-host format in the service URL with the following settings: @@ -403,7 +403,7 @@ webServicePortTls=8443 ### Enable Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -421,7 +421,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker). +If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker.md). ### Start Brokers diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-dcos.md b/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-dcos.md index f5f8d1faa7dbe..952d5f47e30fa 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -If you want to enable all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of +If you want to enable all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-docker.md b/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-docker.md index 64a0939aa787b..8348d78deb237 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-docker.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-docker.md @@ -57,4 +57,4 @@ docker network connect pulsar broker To check whether the containers are successfully connected to the network, enter the `docker network inspect pulsar` command. -For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal). +For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal.md). diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-kubernetes.md index dc7123d000467..1aefc6ad79f71 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-kubernetes.md @@ -6,6 +6,6 @@ original_id: deploy-kubernetes --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-monitoring.md b/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-monitoring.md index ae2ac29943331..f9fe0e0bb97be 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/deploy-monitoring.md @@ -113,7 +113,7 @@ http://$FUNCTIONS_WORKER_ADDRESS:$WORKER_PORT/metrics: You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy). +When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy.md). ## Dashboards @@ -121,7 +121,7 @@ When you collect time series statistics, the major problem is to make sure the n ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager). +The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/develop-schema.md b/site2/website/versioned_docs/version-2.8.2-deprecated/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/develop-schema.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/functions-deploy.md b/site2/website/versioned_docs/version-2.8.2-deprecated/functions-deploy.md index 10a8339edb8d5..9c47208f68fa0 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/functions-deploy.md @@ -9,12 +9,12 @@ original_id: functions-deploy To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface @@ -179,7 +179,7 @@ $ bin/pulsar-admin functions create \ Package management enables version management and simplifies the upgrade and rollback processes for Functions, Sinks, and Sources. When you use the same function, sink and source in different namespaces, you can upload them to a common package management system. -To use [Package management service](admin-api-packages), ensure that the package management service has been enabled in your cluster by setting the following properties in `broker.conf`. +To use [Package management service](admin-api-packages.md), ensure that the package management service has been enabled in your cluster by setting the following properties in `broker.conf`. > Note: Package management service is not enabled by default. diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/functions-develop.md b/site2/website/versioned_docs/version-2.8.2-deprecated/functions-develop.md index 36ea4b4a21073..2e29aa1c47400 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/functions-develop.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/functions-develop.md @@ -1144,7 +1144,7 @@ You can monitor Pulsar Functions that have been deployed with the following meth - Check the metrics provided by Pulsar. - Pulsar Functions expose the metrics that can be collected and used for monitoring the health of **Java, Python, and Go** functions. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar Functions expose the metrics that can be collected and used for monitoring the health of **Java, Python, and Go** functions. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. For the complete list of the function metrics, see [here](reference-metrics.md#pulsar-functions). @@ -1214,7 +1214,7 @@ Currently, the feature is not available in Go. ## Security -If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker). For more details, refer to [Security settings](functions-worker.md#security-settings). +If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings). Pulsar Functions can support the following providers: diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/functions-overview.md b/site2/website/versioned_docs/version-2.8.2-deprecated/functions-overview.md index 25dc6029acf70..816d301e0fd0e 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/functions-overview.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/functions-overview.md @@ -164,7 +164,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/functions-package.md b/site2/website/versioned_docs/version-2.8.2-deprecated/functions-package.md index 82346aa8c5098..a995d5c158877 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/functions-package.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/functions-package.md @@ -15,7 +15,7 @@ Currently, the window function is not available in Python and Go. ## Prerequisite -Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm). +Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm.md). To check whether the Docker image starts, you can use the `docker ps` command. diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/functions-worker.md b/site2/website/versioned_docs/version-2.8.2-deprecated/functions-worker.md index 78f4f19487951..49fc76b30bdaa 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/functions-worker.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/functions-worker.md @@ -4,7 +4,7 @@ title: Deploy and manage functions worker sidebar_label: "Setup: Pulsar Functions Worker" original_id: functions-worker --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -179,7 +179,7 @@ brokerClientTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -199,7 +199,7 @@ authenticationProviders: [ provider1, provider2 ] ``` For *TLS Authentication* provider, follow the example below to add the necessary settings. -See [TLS Authentication](security-tls-authentication) for more details. +See [TLS Authentication](security-tls-authentication.md) for more details. ``` @@ -223,7 +223,7 @@ properties: ``` For *Token Authentication* provider, add necessary settings for `properties` if needed. -See [Token Authentication](security-jwt) for more details. +See [Token Authentication](security-jwt.md) for more details. Note: key files must be DER-encoded ``` @@ -262,7 +262,7 @@ superUserRoles: You can use the public and private key pair that the application configures to perform encryption. Only the consumers with a valid key can decrypt the encrypted messages. -To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption). +To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption.md). We include the relevant configuration information of `CryptoConfig` into `ProducerConfig`. The specific configurable field information about `CryptoConfig` is as follows: diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/getting-started-docker.md b/site2/website/versioned_docs/version-2.8.2-deprecated/getting-started-docker.md index b9f12f0b52688..4f20971d75330 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/getting-started-docker.md @@ -50,8 +50,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -110,7 +110,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/getting-started-helm.md b/site2/website/versioned_docs/version-2.8.2-deprecated/getting-started-helm.md index 65a9fc04fe568..440087c275c05 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/getting-started-helm.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/getting-started-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/getting-started-pulsar.md b/site2/website/versioned_docs/version-2.8.2-deprecated/getting-started-pulsar.md index 11c5e66cb62c2..752590f57b558 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/getting-started-pulsar.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/getting-started-pulsar.md @@ -5,13 +5,13 @@ sidebar_label: "Pulsar 2.0" original_id: pulsar-2.0 --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/getting-started-standalone.md b/site2/website/versioned_docs/version-2.8.2-deprecated/getting-started-standalone.md index 0e40286d71dcb..cea47efd08d4b 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -65,7 +65,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -74,7 +74,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -123,7 +123,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -137,7 +137,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -170,12 +170,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -208,7 +208,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/helm-install.md b/site2/website/versioned_docs/version-2.8.2-deprecated/helm-install.md index edb12260f010e..1f4d5eb69d5dd 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/helm-install.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/helm-install.md @@ -21,7 +21,7 @@ Before deploying Pulsar, you need to prepare your environment. ### Tools -Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools) on your computer. +Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools.md) on your computer. ## Cloud cluster preparation @@ -37,8 +37,8 @@ To create and connect to the Kubernetes cluster, follow the instructions: ## Pulsar deployment -Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy). +Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy.md). ## Pulsar upgrade -To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade). +To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade.md). diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/helm-overview.md b/site2/website/versioned_docs/version-2.8.2-deprecated/helm-overview.md index 9cde85e2d18d5..385d535e319b6 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/helm-overview.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/helm-overview.md @@ -52,9 +52,9 @@ It includes support for: ## Pulsar Helm chart quick start -To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install). +This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install.md). ## Troubleshooting @@ -64,7 +64,7 @@ We have done our best to make these charts as seamless as possible. Occasionally The Apache Pulsar Helm chart contains all required dependencies. -If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm) for your first iteration. +If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm.md) for your first iteration. 1. [Preparation](helm-prepare.md) 2. [Deployment](helm-deploy.md) @@ -82,7 +82,7 @@ helm upgrade apache/pulsar -f pulsar.yaml ``` -For more detailed information, see [Upgrading](helm-upgrade). +For more detailed information, see [Upgrading](helm-upgrade.md). ## Uninstallation diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/helm-prepare.md b/site2/website/versioned_docs/version-2.8.2-deprecated/helm-prepare.md index 95d7e668f9eca..5e9f2f9ef4f68 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/helm-prepare.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/helm-prepare.md @@ -89,4 +89,4 @@ PROJECT= USE_LOCAL_SSD=true LOCAL_SSD_COUNT= ## Next Steps -Continue with the [installation of the chart](helm-deploy) once you have the cluster up and running. +Continue with the [installation of the chart](helm-deploy.md) once you have the cluster up and running. diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/helm-tools.md b/site2/website/versioned_docs/version-2.8.2-deprecated/helm-tools.md index efd61eae32506..6ba89006913b6 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/helm-tools.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/helm-tools.md @@ -25,7 +25,7 @@ You can get Helm from the project's [releases page](https://github.com/helm/helm ### Next steps -Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare). +Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare.md). ## Additional information diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/helm-upgrade.md b/site2/website/versioned_docs/version-2.8.2-deprecated/helm-upgrade.md index e39cf97407355..7d671e6bfb3c1 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/helm-upgrade.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/helm-upgrade.md @@ -20,7 +20,7 @@ You can retrieve your previous `--set` arguments cleanly, with `helm get values To upgrade Apache Pulsar to a newer version, follow these steps: 1. Check the change log for the specific version you would like to upgrade to. -2. Go through [deployment documentation](helm-deploy) step by step. +2. Go through [deployment documentation](helm-deploy.md) step by step. 3. Extract your previous `--set` arguments with the following command. ```bash diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/io-cdc.md b/site2/website/versioned_docs/version-2.8.2-deprecated/io-cdc.md index 20f16ae0d6904..e6e662884826d 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/io-cdc.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/io-cdc.md @@ -13,8 +13,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/io-develop.md b/site2/website/versioned_docs/version-2.8.2-deprecated/io-develop.md index a20206fbf6a06..d6f4f8261ac82 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/io-develop.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/io-develop.md @@ -14,15 +14,15 @@ import TabItem from '@theme/TabItem'; This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -275,7 +275,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -304,7 +304,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). @@ -379,7 +379,7 @@ Pulsar connectors enable you to move data in and out of Pulsar easily. It is imp - Check the metrics provided by Pulsar. - Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. - Set and check your customized metrics. diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/io-overview.md b/site2/website/versioned_docs/version-2.8.2-deprecated/io-overview.md index 0070a015296db..3db5ee34042d3 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/io-overview.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/io-overview.md @@ -158,7 +158,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/io-quickstart.md b/site2/website/versioned_docs/version-2.8.2-deprecated/io-quickstart.md index b8fea1560bdeb..8474c93f51336 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you are able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you are able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However, all the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -109,7 +109,7 @@ This section demonstrates how to connect Pulsar to Cassandra. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink.md). ::: @@ -236,11 +236,11 @@ You can create a configuration file through one of the following methods. ``` -For more information, see [Cassandra sink connector](io-cassandra-sink). +For more information, see [Cassandra sink connector](io-cassandra-sink.md). ### Create a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on them. Run the following command to create a Cassandra sink connector with sink type _cassandra_ and the config file _examples/cassandra-sink.yml_ created previously. @@ -267,7 +267,7 @@ as a Pulsar Function and writes the messages produced in the topic _test_cassand ### Inspect a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * Get the information of a Cassandra sink. @@ -431,7 +431,7 @@ to monitor a connector and perform other operations on it. ### Delete a Cassandra Sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. ```bash @@ -454,7 +454,7 @@ This section demonstrates how to connect Pulsar to PostgreSQL. ::: ->For more information, see [JDBC sink connector](io-jdbc-sink). +>For more information, see [JDBC sink connector](io-jdbc-sink.md). ### Setup a PostgreSQL cluster @@ -614,7 +614,7 @@ In this section, you need to configure a JDBC sink connector. ### Create a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on it. This example creates a sink connector and specifies the desired information. @@ -660,7 +660,7 @@ The sink has been created successfully if the following message appears. ### Inspect a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * List all running JDBC sink(s). @@ -780,7 +780,7 @@ to monitor a connector and perform other operations on it. ### Stop a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to stop a connector and perform other operations on it. ```bash @@ -808,7 +808,7 @@ The sink instance has been stopped successfully if the following message disappe ### Restart a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to restart a connector and perform other operations on it. ```bash @@ -844,7 +844,7 @@ Note that `pulsar-admin sinks localrun options` **runs a sink connector locally* ### Update a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to update a connector and perform other operations on it. This example updates the parallelism of the _pulsar-postgres-jdbc-sink_ sink connector to 2. @@ -912,7 +912,7 @@ The result shows that the parallelism is 2. ### Delete a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. This example deletes the _pulsar-postgres-jdbc-sink_ sink connector. diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/io-use.md b/site2/website/versioned_docs/version-2.8.2-deprecated/io-use.md index b6409d4420ad7..da9ed746c4d37 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/io-use.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/io-use.md @@ -15,7 +15,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/kubernetes-helm.md b/site2/website/versioned_docs/version-2.8.2-deprecated/kubernetes-helm.md index a9f768e6a96b3..ea92a0968cd7d 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/kubernetes-helm.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/kubernetes-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/reference-cli-tools.md b/site2/website/versioned_docs/version-2.8.2-deprecated/reference-cli-tools.md index 3a46361fd8aea..2abd4a47f86ef 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/reference-configuration.md b/site2/website/versioned_docs/version-2.8.2-deprecated/reference-configuration.md index 5f2212650e9ea..e761161e99cea 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/reference-configuration.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/reference-configuration.md @@ -8,7 +8,7 @@ original_id: reference-configuration -You can manage Pulsar configuration by configuration files in the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a Pulsar [installation](getting-started-standalone). +You can manage Pulsar configuration by configuration files in the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a Pulsar [installation](getting-started-standalone.md). - [BookKeeper](#bookkeeper) - [Broker](#broker) diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/schema-get-started.md b/site2/website/versioned_docs/version-2.8.2-deprecated/schema-get-started.md index 97e270586a26d..afacb0fa51f2e 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/schema-get-started.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/schema-get-started.md @@ -17,7 +17,7 @@ Applications typically adopt one of the following approaches to guarantee type s #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ### Client-side approach diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/schema-manage.md b/site2/website/versioned_docs/version-2.8.2-deprecated/schema-manage.md index ec05a0f2babea..4607c3b5dcae1 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/schema-manage.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/schema-manage.md @@ -160,7 +160,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/schema-understand.md b/site2/website/versioned_docs/version-2.8.2-deprecated/schema-understand.md index 153d96e6e71cb..a86b02add435e 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/schema-understand.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/schema-understand.md @@ -459,7 +459,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```java diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/security-athenz.md b/site2/website/versioned_docs/version-2.8.2-deprecated/security-athenz.md index 947c3f470be46..8a39fe25316d0 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/security-athenz.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/security-athenz.md @@ -76,7 +76,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/security-authorization.md b/site2/website/versioned_docs/version-2.8.2-deprecated/security-authorization.md index cc9c03d5fc1dd..9cfd7c8c203f6 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/security-authorization.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/security-authorization.md @@ -27,7 +27,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -59,7 +59,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -87,7 +87,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication @@ -117,7 +117,7 @@ PulsarAdmin admin = PulsarAdmin.builder() When a client is identified with multiple roles in a token (the type of role claim in the token is an array) during the authentication process, Pulsar supports to check the permissions of all the roles and further authorize the client as long as one of its roles has the required permissions. > **Note**
    -> This authorization method is only compatible with [JWT authentication](security-jwt). +> This authorization method is only compatible with [JWT authentication](security-jwt.md). To enable this authorization method, configure the authorization provider as `MultiRolesTokenAuthorizationProvider` in the `conf/broker.conf` file. diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/security-bouncy-castle.md b/site2/website/versioned_docs/version-2.8.2-deprecated/security-bouncy-castle.md index abc25c71b8ae2..be937055d8e31 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/security-bouncy-castle.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/security-bouncy-castle.md @@ -16,7 +16,7 @@ In Pulsar, security and crypto have dependencies on BouncyCastle Jars. For the d `Bouncy Castle` provides both [FIPS](https://www.bouncycastle.org/fips_faq.html) and non-FIPS version. But in a JVM, you can not include both of the 2 versions, and you need to exclude the current version before include the other. -In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. +In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption.md). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. ## How BouncyCastle modules packaged in Pulsar diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/security-extending.md b/site2/website/versioned_docs/version-2.8.2-deprecated/security-extending.md index c088e3a2c1c2c..e7484453b8beb 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/security-extending.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/security-extending.md @@ -9,7 +9,7 @@ Pulsar provides a way to use custom authentication and authorization mechanisms. ## Authentication -Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview). +Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview.md). You can use a custom authentication mechanism by providing the implementation in the form of two plugins. One plugin is for the Client library and the other plugin is for the Pulsar Proxy and/or Pulsar Broker to validate the credentials. diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/security-jwt.md b/site2/website/versioned_docs/version-2.8.2-deprecated/security-jwt.md index 05e641ed67c17..1fa65b7c27f60 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/security-jwt.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/security-jwt.md @@ -32,11 +32,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -305,7 +305,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/security-kerberos.md b/site2/website/versioned_docs/version-2.8.2-deprecated/security-kerberos.md index 670586a9d1691..c49fa3bea1fce 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/security-kerberos.md @@ -373,7 +373,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/security-overview.md b/site2/website/versioned_docs/version-2.8.2-deprecated/security-overview.md index f8e8eac98f805..227dd5b5d4fc3 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/security-overview.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/security-overview.md @@ -21,7 +21,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/security-tls-authentication.md b/site2/website/versioned_docs/version-2.8.2-deprecated/security-tls-authentication.md index f3c32a05d9751..85d2240f41306 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/security-tls-authentication.md @@ -7,9 +7,9 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -103,7 +103,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -123,7 +123,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/security-tls-keystore.md b/site2/website/versioned_docs/version-2.8.2-deprecated/security-tls-keystore.md index 05b919cc15932..2d80782ba88d3 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/security-tls-keystore.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/security-tls-keystore.md @@ -7,7 +7,7 @@ original_id: security-tls-keystore ## Overview -Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication) between clients and Apache Pulsar service. +Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication.md) between clients and Apache Pulsar service. By default it uses PEM format file configuration. This page tries to describe use [KeyStore](https://en.wikipedia.org/wiki/Java_KeyStore) type configure for TLS. @@ -185,7 +185,7 @@ This is similar to [TLS encryption configuing for client with PEM type](security For a a minimal configuration, user need to provide the TrustStore information. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties @@ -278,7 +278,7 @@ webSocketServiceEnabled=false Besides the TLS encryption configuring. The main work is configuring the KeyStore, which contains a valid CN as client role, for client. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/security-tls-transport.md b/site2/website/versioned_docs/version-2.8.2-deprecated/security-tls-transport.md index 92e8dc3c3d280..2cad17a78c350 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,9 +27,9 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). -`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ## Create TLS certificates @@ -130,7 +130,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Configure broker -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -202,7 +202,7 @@ The examples below show that hostname verification is disabled for the CLI tools ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/security-token-admin.md b/site2/website/versioned_docs/version-2.8.2-deprecated/security-token-admin.md index 1a438dc7923ee..a265f6320d28f 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/security-token-admin.md @@ -163,7 +163,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.8.2-deprecated/sql-deployment-configurations.md index 67f7ab17c77d0..02d8bc78f6cb9 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/sql-deployment-configurations.md @@ -64,7 +64,7 @@ Since Pulsar SQL is powered by [Trino (formerly Presto SQL)](https://trino.io), :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/sql-overview.md b/site2/website/versioned_docs/version-2.8.2-deprecated/sql-overview.md index 4a4d5f01b0bc5..0235d0256c5f1 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/sql-overview.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/sql-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Overview" original_id: sql-overview --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL)](https://trino.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL.md)](https://trino.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/standalone-docker.md b/site2/website/versioned_docs/version-2.8.2-deprecated/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/standalone.md b/site2/website/versioned_docs/version-2.8.2-deprecated/standalone.md index d114c017d7c00..25afa11a91b11 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/standalone.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -65,7 +65,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -74,7 +74,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -123,7 +123,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -137,7 +137,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -170,12 +170,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -208,7 +208,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.8.2-deprecated/tiered-storage-overview.md b/site2/website/versioned_docs/version-2.8.2-deprecated/tiered-storage-overview.md index 7c6528ce2b36c..c635034f463b4 100644 --- a/site2/website/versioned_docs/version-2.8.2-deprecated/tiered-storage-overview.md +++ b/site2/website/versioned_docs/version-2.8.2-deprecated/tiered-storage-overview.md @@ -13,9 +13,9 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws). + - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws.md). - - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs). + - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs.md). ::: @@ -25,7 +25,7 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem). + For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem.md). ::: diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/admin-api-brokers.md b/site2/website/versioned_docs/version-2.8.3-deprecated/admin-api-brokers.md index 3effc4a192326..aea8f70f87a7d 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/admin-api-brokers.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/admin-api-brokers.md @@ -28,7 +28,7 @@ Pulsar brokers consist of two components: [Brokers](reference-terminology.md#broker) can be managed via: -* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`brokers`](reference-pulsar-admin.md#brokers) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/brokers` endpoint of the admin {@inject: rest:REST:/} API * The `brokers` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin.html} object in the [Java API](client-libraries-java.md) @@ -176,7 +176,7 @@ One way to configure a Pulsar [broker](reference-terminology.md#broker) is to su But since all broker configuration in Pulsar is stored in ZooKeeper, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values. -* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. +* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. * In the Pulsar admin {@inject: rest:REST:/} API, dynamic configuration is managed through the `/admin/v2/brokers/configuration` endpoint. ### Update dynamic configuration diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/admin-api-clusters.md b/site2/website/versioned_docs/version-2.8.3-deprecated/admin-api-clusters.md index c4b404fc8e680..ce085345640a4 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/admin-api-clusters.md @@ -26,7 +26,7 @@ servers (aka [bookies](reference-terminology.md#bookie)), and a [ZooKeeper](http Clusters can be managed via: -* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin) tool +* The [`clusters`](reference-pulsar-admin.md#clusters) command of the [`pulsar-admin`](reference-pulsar-admin.md) tool * The `/admin/v2/clusters` endpoint of the admin {@inject: rest:REST:/} API * The `clusters` method of the {@inject: javadoc:PulsarAdmin:/admin/org/apache/pulsar/client/admin/PulsarAdmin} object in the [Java API](client-libraries-java.md) @@ -89,7 +89,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -113,11 +113,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block > - For the latest and complete information about `Java admin API`, including classes, methods, descriptions, and more, see [Java admin API doc](https://pulsar.apache.org/api/admin/). -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -118,7 +118,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-cgo.md b/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-cgo.md index c79f7bb965473..f352f942b7714 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-cgo.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-cgo.md @@ -24,7 +24,7 @@ Currently, the following Go clients are maintained in two repositories. ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Install go package @@ -57,7 +57,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -75,7 +75,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -236,14 +236,14 @@ Parameter | Description | Default `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | `Properties`| Attach a set of application defined properties to the producer. This properties will be visible in the topic stats | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | Set the number of max pending messages across all the partitions. This setting will be used to lower the max pending messages for each partition `MaxPendingMessages(int)`, if the total exceeds the configured value.| `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | `Batching` | Control whether automatic batching of messages is enabled for the producer. | false `BatchingMaxPublishDelay` | Set the time period within which the messages sent will be batched (default: 1ms) if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or until | 1ms `BatchingMaxMessages` | Set the maximum number of messages permitted in a batch. (default: 1000) If set to a value greater than 1, messages will be queued until this threshold is reached or batch interval has elapsed | 1000 @@ -512,7 +512,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-cpp.md index deebe13bada6d..08622b9e83071 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-cpp.md @@ -372,7 +372,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema -This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started). +This section describes some examples about schema. For more information about schema, see [Pulsar schema](schema-get-started.md). ### Create producer with Avro schema diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-dotnet.md b/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-dotnet.md index 4e0afe3d2ff6b..fbec5e473be69 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-dotnet.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-dotnet.md @@ -170,7 +170,7 @@ var client = PulsarClient.Builder() Currently, the Pulsar C# client supports the TLS (Transport Layer Security) and JWT (JSON Web Token) authentication. -If you have followed [Authentication using TLS](security-tls-authentication), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: +If you have followed [Authentication using TLS](security-tls-authentication.md), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: 1. Create an unencrypted and password-less pfx file. diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-go.md b/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-go.md index fbb2573aa5db8..6281b03dd8c80 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-go.md @@ -35,7 +35,7 @@ import "github.com/apache/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -61,7 +61,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -844,7 +844,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker @@ -864,7 +864,7 @@ opts := pulsar.ClientOptions{ ## OAuth2 authentication -To use [OAuth2 authentication](security-oauth2), you'll need to configure your client to perform the following operations. +To use [OAuth2 authentication](security-oauth2.md), you'll need to configure your client to perform the following operations. This example shows how to configure OAuth2 authentication. ```go diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-java.md b/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-java.md index e57c83f7963f4..0ff9a22936faf 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-java.md @@ -5,7 +5,7 @@ sidebar_label: "Java" original_id: client-libraries-java --- -You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview). The current version of the Java client is **@pulsar:version@**. +You can use Pulsar Java client to create Java [producer](#producer), [consumer](#consumer), and [readers](#reader-interface) of messages and to perform [administrative tasks](admin-api-overview.md). The current version of the Java client is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), and [reader](#reader) of a Java client are thread-safe. @@ -14,10 +14,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Includes both `pulsar-client` and `pulsar-client-admin`

    Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.

    In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -57,7 +57,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -83,7 +83,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -114,7 +114,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -240,7 +240,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned) cookbook. +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics](cookbooks-partitioned.md) cookbook. ### Async send @@ -827,7 +827,7 @@ Total hash range size is 65536, so the max end of the range should be less than ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producers) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -945,7 +945,7 @@ For example of ProtobufNativeSchema, see [`SchemaDefinition` in `Complex type`]( ## Authentication -Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2). You can use the Pulsar Java client with all of them. +Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2.md). You can use the Pulsar Java client with all of them. ### TLS Authentication @@ -972,7 +972,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` @@ -1009,7 +1009,7 @@ PulsarClient client = PulsarClient.builder() ### Oauth2 -The following example shows how to use [Oauth2](security-oauth2) as an authentication provider for the Pulsar Java client. +The following example shows how to use [Oauth2](security-oauth2.md) as an authentication provider for the Pulsar Java client. You can use the factory method to configure authentication for Pulsar Java client. diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-node.md b/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-node.md index 3cb743bc300c5..8031d287c6f2a 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-node.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-node.md @@ -48,7 +48,7 @@ Also, this library works only in Node.js 10.x or later because it uses the [`nod ::: ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here is an example for `localhost`: @@ -66,7 +66,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication), the URL looks like this: +If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication.md), the URL looks like this: ```http @@ -101,7 +101,7 @@ The following configurable parameters are available for Pulsar clients: | Parameter | Description | Default | | :-------- | :---------- | :------ | | `serviceUrl` | The connection URL for the Pulsar cluster. See [above](#connection-urls) for more info. | | -| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication) for more info. | | +| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication.md) for more info. | | | `operationTimeoutSeconds` | The timeout for Node.js client operations (creating producers, subscribing to and unsubscribing from [topics](reference-terminology.md#topic)). Retries occur until this threshold is reached, at which point the operation fails. | 30 | | `ioThreads` | The number of threads to use for handling connections to Pulsar [brokers](reference-terminology.md#broker). | 1 | | `messageListenerThreads` | The number of threads used by message listeners ([consumers](#consumers) and [readers](#readers)). | 1 | @@ -154,7 +154,7 @@ Pulsar Node.js producers have the following methods available: | :-------- | :---------- | :------ | | `topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer publishes messages. The topic format is `` or `//`. For example, `sample/ns1/my-topic`. | | | `producerName` | A name for the producer. If you do not explicitly assign a name, Pulsar automatically generates a globally unique name. If you choose to explicitly assign a name, it needs to be unique across *all* Pulsar clusters, otherwise the creation operation throws an error. | | -| `sendTimeoutMs` | When publishing a message to a topic, the producer waits for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error is thrown. If you set `sendTimeoutMs` to -1, the timeout is set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30000 | +| `sendTimeoutMs` | When publishing a message to a topic, the producer waits for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error is thrown. If you set `sendTimeoutMs` to -1, the timeout is set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30000 | | `initialSequenceId` | The initial sequence ID of the message. When producer send message, add sequence ID to message. The ID is increased each time to send. | | | `maxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `send` method fails *unless* `blockIfQueueFull` is set to `true`. | 1000 | | `maxPendingMessagesAcrossPartitions` | The maximum size of the sum of partition's pending queue. | 50000 | diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-python.md b/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-python.md index 7ebc6bec64e5d..f30cf55387d92 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [`python` subdirectory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-websocket.md index 913b6cf5c82a9..e2da41f0461b4 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/client-libraries-websocket.md @@ -5,14 +5,14 @@ sidebar_label: "WebSocket" original_id: client-libraries-websocket --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-architecture-overview.md index 0c7d44260ddeb..f3e75c3e307e0 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-architecture-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Architecture" original_id: concepts-architecture-overview --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -17,20 +17,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers. The producers connect to the brokers to publish messages and the consumers connect to the brokers to consume the messages. -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -40,9 +40,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -136,17 +136,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions in the [Deploying a Pulsar instance](deploy-bare-metal.md#service-discovery-setup) guide. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -154,7 +154,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-authentication.md b/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-authentication.md index b375ecb16a2d8..f6307890c904a 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-authentication.md @@ -5,5 +5,5 @@ sidebar_label: "Authentication and Authorization" original_id: concepts-authentication --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-clients.md b/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-clients.md index b68f76a2d8b08..4040624f7d636 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-clients.md @@ -5,12 +5,12 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -38,7 +38,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-messaging.md b/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-messaging.md index 4be25256b4b36..b76728f109b5d 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-messaging.md @@ -21,8 +21,8 @@ Messages are the basic "unit" of Pulsar. The following table lists the component Component | Description :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer who produces the message. If you do not specify a producer name, the default name is used. Sequence ID | Each Pulsar message belongs to an ordered sequence on its topic. The sequence ID of the message is its order in that sequence. @@ -50,7 +50,7 @@ The default size of a message is 5 MB. You can configure the max size of a messa ``` -> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol). +> For more information on Pulsar message contents, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -147,7 +147,7 @@ Messages are received from [brokers](reference-terminology.md#broker) either syn ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -384,7 +384,7 @@ A [subscription](#subscriptions) can have one or more consumers. When a consumer #### When to use -By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry). +By default, messages of a topic without any durable subscriptions are marked as deleted. If you want to prevent the messages being marked as deleted, you can create a durable subscription for this topic. In this case, only acknowledged messages are marked as deleted. For more information, see [message retention and expiry](cookbooks-retention-expiry.md). #### How to use @@ -488,7 +488,7 @@ Decisions about routing and subscription types can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription types work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -500,7 +500,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -538,7 +538,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -625,7 +625,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -648,12 +648,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication.md). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-multi-tenancy.md index 730458c904618..93a59557b2efc 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-multi-tenancy.md @@ -5,7 +5,7 @@ sidebar_label: "Multi Tenancy" original_id: concepts-multi-tenancy --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -21,7 +21,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -29,7 +29,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-overview.md b/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-overview.md index c1262b38daf16..e8a2f4b9d321a 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription types](concepts-messaging.md#subscription-types) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-replication.md b/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-replication.md index 6e23962e99771..799f0eb4d92c6 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-replication.md @@ -5,5 +5,5 @@ sidebar_label: "Geo Replication" original_id: concepts-replication --- -Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that. +Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo.md) in Pulsar enables you to do that. diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-tiered-storage.md index 0b45b0a2ed501..f6988e53a8cd4 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-topic-compaction.md index e402c965384fd..34b7ed7fbbd31 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/concepts-topic-compaction.md @@ -7,7 +7,7 @@ original_id: concepts-topic-compaction Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -24,7 +24,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.8.3-deprecated/cookbooks-compaction.md index 6fcbc197f9ecb..dfa314727241a 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.8.3-deprecated/cookbooks-non-persistent.md index 391569a8cbc6b..178301e86eb8d 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/cookbooks-non-persistent.md @@ -41,7 +41,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -55,7 +55,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.8.3-deprecated/cookbooks-partitioned.md index 7882fb9025312..fb9ac354cc6d6 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/cookbooks-partitioned.md @@ -4,4 +4,4 @@ title: Partitioned topics sidebar_label: "Partitioned Topics" original_id: cookbooks-partitioned --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.8.3-deprecated/cookbooks-retention-expiry.md index af4be6bb2d9c6..192fbe6c53277 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/cookbooks-retention-expiry.md @@ -22,7 +22,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-aws.md b/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-aws.md index 1c8f1a08d7249..662e99ceb9b09 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-aws.md @@ -5,7 +5,7 @@ sidebar_label: "Amazon Web Services" original_id: deploy-aws --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -210,7 +210,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md), edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-bare-metal-multi-cluster.md index dacd571365aa2..1b23eea07a20b 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-bare-metal-multi-cluster.md @@ -9,17 +9,17 @@ original_id: deploy-bare-metal-multi-cluster 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using it in a startup or on a single team, you had better opt for a single cluster. For instructions on deploying a single cluster, -see the guide [here](deploy-bare-metal). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo). Deploying a multi-cluster Pulsar instance involves the following basic steps: +A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md). Deploying a multi-cluster Pulsar instance involves the following basic steps: * Deploying two separate [ZooKeeper](#deploy-zookeeper) quorums: a [local](#deploy-local-zookeeper) quorum for each cluster in the instance and a [configuration store](#configuration-store) quorum for instance-wide tasks * Initializing [cluster metadata](#cluster-metadata-initialization) for each cluster @@ -29,7 +29,7 @@ A Pulsar *instance* consists of multiple Pulsar clusters working in unison. You If you want to deploy a single Pulsar cluster, see [Clusters and Brokers](getting-started-standalone.md#start-the-cluster). > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes on [Google Kubernetes Engine](deploy-kubernetes#pulsar-on-google-kubernetes-engine) and on [Amazon Web Services](deploy-kubernetes#pulsar-on-amazon-web-services). ## System requirement @@ -74,7 +74,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -252,7 +252,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -371,17 +371,17 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. Pulsar provides a built-in service discovery mechanism that you can set up using the instructions [immediately below](#service-discovery-setup). -You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can also use your own service discovery system if you want. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > #### Service discovery already provided by many scheduling systems -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ### Service discovery setup -The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol) of Pulsar. +The service discovery mechanism that included with Pulsar maintains a list of active brokers, which stored in ZooKeeper, and supports lookup using HTTP and also the [binary protocol](developing-binary-protocol.md) of Pulsar. To get started setting up the built-in service of discovery of Pulsar, you need to change a few parameters in the [`conf/discovery.conf`](reference-configuration.md#service-discovery) configuration file. Set the [`zookeeperServers`](reference-configuration.md#service-discovery-zookeeperServers) parameter to the ZooKeeper quorum connection string of the cluster and the [`configurationStoreServers`](reference-configuration.md#service-discovery-configurationStoreServers) setting to the [configuration store](reference-terminology.md#configuration-store) quorum connection string. @@ -406,7 +406,7 @@ $ bin/pulsar-daemon start discovery ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-bare-metal.md index f856f9f8b1ca6..bdd05f24f2566 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-bare-metal.md @@ -9,13 +9,13 @@ original_id: deploy-bare-metal 1. Single-cluster Pulsar installations should be sufficient for all but the most ambitious use cases. If you are interested in experimenting with Pulsar or using Pulsar in a startup or on a single team, it is simplest to opt for a single cluster. If you do need to run a multi-cluster Pulsar instance, -see the guide [here](deploy-bare-metal-multi-cluster). -2. If you want to use all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` +see the guide [here](deploy-bare-metal-multi-cluster.md). +2. If you want to use all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you need to download `apache-pulsar-io-connectors` package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you -have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` +have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders` package and install `apache-pulsar-offloaders` under `offloaders` directory in the pulsar directory on every broker node. For more details of how to configure -this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -107,7 +107,7 @@ The extracted directory contains the following subdirectories: Directory | Contains :---------|:-------- -`bin` |[command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` |[command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `data` | The data storage directory that ZooKeeper and BookKeeper use `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -261,9 +261,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (you had better not use a different port). -`--web-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). +`--web-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should not use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (you had better not use a different port). -`--broker-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). +`--broker-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). > If you do not have a DNS server, you can use multi-host format in the service URL with the following settings: @@ -403,7 +403,7 @@ webServicePortTls=8443 ### Enable Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -421,7 +421,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker). +If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker.md). ### Start Brokers diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-dcos.md b/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-dcos.md index f5f8d1faa7dbe..952d5f47e30fa 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -If you want to enable all builtin [Pulsar IO](io-overview) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of +If you want to enable all builtin [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, you can choose to use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-docker.md b/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-docker.md index 64a0939aa787b..8348d78deb237 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-docker.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-docker.md @@ -57,4 +57,4 @@ docker network connect pulsar broker To check whether the containers are successfully connected to the network, enter the `docker network inspect pulsar` command. -For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal). +For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal.md). diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-kubernetes.md index dc7123d000467..1aefc6ad79f71 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-kubernetes.md @@ -6,6 +6,6 @@ original_id: deploy-kubernetes --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-monitoring.md b/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-monitoring.md index ae2ac29943331..f9fe0e0bb97be 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/deploy-monitoring.md @@ -113,7 +113,7 @@ http://$FUNCTIONS_WORKER_ADDRESS:$WORKER_PORT/metrics: You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy). +When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy.md). ## Dashboards @@ -121,7 +121,7 @@ When you collect time series statistics, the major problem is to make sure the n ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager). +The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/develop-schema.md b/site2/website/versioned_docs/version-2.8.3-deprecated/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/develop-schema.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/functions-deploy.md b/site2/website/versioned_docs/version-2.8.3-deprecated/functions-deploy.md index 10a8339edb8d5..9c47208f68fa0 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/functions-deploy.md @@ -9,12 +9,12 @@ original_id: functions-deploy To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface @@ -179,7 +179,7 @@ $ bin/pulsar-admin functions create \ Package management enables version management and simplifies the upgrade and rollback processes for Functions, Sinks, and Sources. When you use the same function, sink and source in different namespaces, you can upload them to a common package management system. -To use [Package management service](admin-api-packages), ensure that the package management service has been enabled in your cluster by setting the following properties in `broker.conf`. +To use [Package management service](admin-api-packages.md), ensure that the package management service has been enabled in your cluster by setting the following properties in `broker.conf`. > Note: Package management service is not enabled by default. diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/functions-develop.md b/site2/website/versioned_docs/version-2.8.3-deprecated/functions-develop.md index 36ea4b4a21073..2e29aa1c47400 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/functions-develop.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/functions-develop.md @@ -1144,7 +1144,7 @@ You can monitor Pulsar Functions that have been deployed with the following meth - Check the metrics provided by Pulsar. - Pulsar Functions expose the metrics that can be collected and used for monitoring the health of **Java, Python, and Go** functions. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar Functions expose the metrics that can be collected and used for monitoring the health of **Java, Python, and Go** functions. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. For the complete list of the function metrics, see [here](reference-metrics.md#pulsar-functions). @@ -1214,7 +1214,7 @@ Currently, the feature is not available in Go. ## Security -If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker). For more details, refer to [Security settings](functions-worker.md#security-settings). +If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings). Pulsar Functions can support the following providers: diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/functions-overview.md b/site2/website/versioned_docs/version-2.8.3-deprecated/functions-overview.md index 25dc6029acf70..816d301e0fd0e 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/functions-overview.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/functions-overview.md @@ -164,7 +164,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/functions-package.md b/site2/website/versioned_docs/version-2.8.3-deprecated/functions-package.md index 82346aa8c5098..a995d5c158877 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/functions-package.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/functions-package.md @@ -15,7 +15,7 @@ Currently, the window function is not available in Python and Go. ## Prerequisite -Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm). +Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm.md). To check whether the Docker image starts, you can use the `docker ps` command. diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/functions-worker.md b/site2/website/versioned_docs/version-2.8.3-deprecated/functions-worker.md index 78f4f19487951..49fc76b30bdaa 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/functions-worker.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/functions-worker.md @@ -4,7 +4,7 @@ title: Deploy and manage functions worker sidebar_label: "Setup: Pulsar Functions Worker" original_id: functions-worker --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -179,7 +179,7 @@ brokerClientTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -199,7 +199,7 @@ authenticationProviders: [ provider1, provider2 ] ``` For *TLS Authentication* provider, follow the example below to add the necessary settings. -See [TLS Authentication](security-tls-authentication) for more details. +See [TLS Authentication](security-tls-authentication.md) for more details. ``` @@ -223,7 +223,7 @@ properties: ``` For *Token Authentication* provider, add necessary settings for `properties` if needed. -See [Token Authentication](security-jwt) for more details. +See [Token Authentication](security-jwt.md) for more details. Note: key files must be DER-encoded ``` @@ -262,7 +262,7 @@ superUserRoles: You can use the public and private key pair that the application configures to perform encryption. Only the consumers with a valid key can decrypt the encrypted messages. -To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption). +To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption.md). We include the relevant configuration information of `CryptoConfig` into `ProducerConfig`. The specific configurable field information about `CryptoConfig` is as follows: diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/getting-started-docker.md b/site2/website/versioned_docs/version-2.8.3-deprecated/getting-started-docker.md index b9f12f0b52688..4f20971d75330 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/getting-started-docker.md @@ -50,8 +50,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -110,7 +110,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/getting-started-helm.md b/site2/website/versioned_docs/version-2.8.3-deprecated/getting-started-helm.md index 22ab38b51878b..c76b1d8ff1756 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/getting-started-helm.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/getting-started-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -406,7 +406,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/getting-started-pulsar.md b/site2/website/versioned_docs/version-2.8.3-deprecated/getting-started-pulsar.md index 11c5e66cb62c2..752590f57b558 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/getting-started-pulsar.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/getting-started-pulsar.md @@ -5,13 +5,13 @@ sidebar_label: "Pulsar 2.0" original_id: pulsar-2.0 --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/getting-started-standalone.md b/site2/website/versioned_docs/version-2.8.3-deprecated/getting-started-standalone.md index 0e40286d71dcb..cea47efd08d4b 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -65,7 +65,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -74,7 +74,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -123,7 +123,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -137,7 +137,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -170,12 +170,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -208,7 +208,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/helm-install.md b/site2/website/versioned_docs/version-2.8.3-deprecated/helm-install.md index edb12260f010e..1f4d5eb69d5dd 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/helm-install.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/helm-install.md @@ -21,7 +21,7 @@ Before deploying Pulsar, you need to prepare your environment. ### Tools -Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools) on your computer. +Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools.md) on your computer. ## Cloud cluster preparation @@ -37,8 +37,8 @@ To create and connect to the Kubernetes cluster, follow the instructions: ## Pulsar deployment -Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy). +Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy.md). ## Pulsar upgrade -To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade). +To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade.md). diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/helm-overview.md b/site2/website/versioned_docs/version-2.8.3-deprecated/helm-overview.md index 9cde85e2d18d5..385d535e319b6 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/helm-overview.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/helm-overview.md @@ -52,9 +52,9 @@ It includes support for: ## Pulsar Helm chart quick start -To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +To get up and run with these charts as fast as possible, in a **non-production** use case, we provide a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install). +This guide walks the user through deploying these charts with default values and features, but *does not* meet production ready requirements. To deploy these charts into production under sustained load, follow the complete [Installation Guide](helm-install.md). ## Troubleshooting @@ -64,7 +64,7 @@ We have done our best to make these charts as seamless as possible. Occasionally The Apache Pulsar Helm chart contains all required dependencies. -If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm) for your first iteration. +If you deploy a PoC for testing, we strongly suggest you follow our [Quick Start Guide](getting-started-helm.md) for your first iteration. 1. [Preparation](helm-prepare.md) 2. [Deployment](helm-deploy.md) @@ -82,7 +82,7 @@ helm upgrade apache/pulsar -f pulsar.yaml ``` -For more detailed information, see [Upgrading](helm-upgrade). +For more detailed information, see [Upgrading](helm-upgrade.md). ## Uninstallation diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/helm-prepare.md b/site2/website/versioned_docs/version-2.8.3-deprecated/helm-prepare.md index 95d7e668f9eca..5e9f2f9ef4f68 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/helm-prepare.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/helm-prepare.md @@ -89,4 +89,4 @@ PROJECT= USE_LOCAL_SSD=true LOCAL_SSD_COUNT= ## Next Steps -Continue with the [installation of the chart](helm-deploy) once you have the cluster up and running. +Continue with the [installation of the chart](helm-deploy.md) once you have the cluster up and running. diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/helm-tools.md b/site2/website/versioned_docs/version-2.8.3-deprecated/helm-tools.md index efd61eae32506..6ba89006913b6 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/helm-tools.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/helm-tools.md @@ -25,7 +25,7 @@ You can get Helm from the project's [releases page](https://github.com/helm/helm ### Next steps -Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare). +Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare.md). ## Additional information diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/helm-upgrade.md b/site2/website/versioned_docs/version-2.8.3-deprecated/helm-upgrade.md index e39cf97407355..7d671e6bfb3c1 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/helm-upgrade.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/helm-upgrade.md @@ -20,7 +20,7 @@ You can retrieve your previous `--set` arguments cleanly, with `helm get values To upgrade Apache Pulsar to a newer version, follow these steps: 1. Check the change log for the specific version you would like to upgrade to. -2. Go through [deployment documentation](helm-deploy) step by step. +2. Go through [deployment documentation](helm-deploy.md) step by step. 3. Extract your previous `--set` arguments with the following command. ```bash diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/io-cdc.md b/site2/website/versioned_docs/version-2.8.3-deprecated/io-cdc.md index 20f16ae0d6904..e6e662884826d 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/io-cdc.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/io-cdc.md @@ -13,8 +13,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/io-develop.md b/site2/website/versioned_docs/version-2.8.3-deprecated/io-develop.md index a20206fbf6a06..d6f4f8261ac82 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/io-develop.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/io-develop.md @@ -14,15 +14,15 @@ import TabItem from '@theme/TabItem'; This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -275,7 +275,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -304,7 +304,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). @@ -379,7 +379,7 @@ Pulsar connectors enable you to move data in and out of Pulsar easily. It is imp - Check the metrics provided by Pulsar. - Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. - Set and check your customized metrics. diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/io-overview.md b/site2/website/versioned_docs/version-2.8.3-deprecated/io-overview.md index 0070a015296db..3db5ee34042d3 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/io-overview.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/io-overview.md @@ -158,7 +158,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/io-quickstart.md b/site2/website/versioned_docs/version-2.8.3-deprecated/io-quickstart.md index b8fea1560bdeb..8474c93f51336 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you are able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you are able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However, all the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -109,7 +109,7 @@ This section demonstrates how to connect Pulsar to Cassandra. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink.md). ::: @@ -236,11 +236,11 @@ You can create a configuration file through one of the following methods. ``` -For more information, see [Cassandra sink connector](io-cassandra-sink). +For more information, see [Cassandra sink connector](io-cassandra-sink.md). ### Create a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on them. Run the following command to create a Cassandra sink connector with sink type _cassandra_ and the config file _examples/cassandra-sink.yml_ created previously. @@ -267,7 +267,7 @@ as a Pulsar Function and writes the messages produced in the topic _test_cassand ### Inspect a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * Get the information of a Cassandra sink. @@ -431,7 +431,7 @@ to monitor a connector and perform other operations on it. ### Delete a Cassandra Sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. ```bash @@ -454,7 +454,7 @@ This section demonstrates how to connect Pulsar to PostgreSQL. ::: ->For more information, see [JDBC sink connector](io-jdbc-sink). +>For more information, see [JDBC sink connector](io-jdbc-sink.md). ### Setup a PostgreSQL cluster @@ -614,7 +614,7 @@ In this section, you need to configure a JDBC sink connector. ### Create a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on it. This example creates a sink connector and specifies the desired information. @@ -660,7 +660,7 @@ The sink has been created successfully if the following message appears. ### Inspect a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * List all running JDBC sink(s). @@ -780,7 +780,7 @@ to monitor a connector and perform other operations on it. ### Stop a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to stop a connector and perform other operations on it. ```bash @@ -808,7 +808,7 @@ The sink instance has been stopped successfully if the following message disappe ### Restart a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to restart a connector and perform other operations on it. ```bash @@ -844,7 +844,7 @@ Note that `pulsar-admin sinks localrun options` **runs a sink connector locally* ### Update a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to update a connector and perform other operations on it. This example updates the parallelism of the _pulsar-postgres-jdbc-sink_ sink connector to 2. @@ -912,7 +912,7 @@ The result shows that the parallelism is 2. ### Delete a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. This example deletes the _pulsar-postgres-jdbc-sink_ sink connector. diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/io-use.md b/site2/website/versioned_docs/version-2.8.3-deprecated/io-use.md index b6409d4420ad7..da9ed746c4d37 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/io-use.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/io-use.md @@ -15,7 +15,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/reference-cli-tools.md b/site2/website/versioned_docs/version-2.8.3-deprecated/reference-cli-tools.md index 3a46361fd8aea..2abd4a47f86ef 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/reference-configuration.md b/site2/website/versioned_docs/version-2.8.3-deprecated/reference-configuration.md index 372e1583c3e61..c53a2fb746652 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/reference-configuration.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/reference-configuration.md @@ -8,7 +8,7 @@ original_id: reference-configuration -You can manage Pulsar configuration by configuration files in the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a Pulsar [installation](getting-started-standalone). +You can manage Pulsar configuration by configuration files in the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a Pulsar [installation](getting-started-standalone.md). - [BookKeeper](#bookkeeper) - [Broker](#broker) diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/schema-get-started.md b/site2/website/versioned_docs/version-2.8.3-deprecated/schema-get-started.md index 97e270586a26d..afacb0fa51f2e 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/schema-get-started.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/schema-get-started.md @@ -17,7 +17,7 @@ Applications typically adopt one of the following approaches to guarantee type s #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ### Client-side approach diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/schema-manage.md b/site2/website/versioned_docs/version-2.8.3-deprecated/schema-manage.md index ec05a0f2babea..4607c3b5dcae1 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/schema-manage.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/schema-manage.md @@ -160,7 +160,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/schema-understand.md b/site2/website/versioned_docs/version-2.8.3-deprecated/schema-understand.md index 153d96e6e71cb..a86b02add435e 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/schema-understand.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/schema-understand.md @@ -459,7 +459,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```java diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/security-athenz.md b/site2/website/versioned_docs/version-2.8.3-deprecated/security-athenz.md index 947c3f470be46..8a39fe25316d0 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/security-athenz.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/security-athenz.md @@ -76,7 +76,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/security-authorization.md b/site2/website/versioned_docs/version-2.8.3-deprecated/security-authorization.md index e6785873f0e10..7ac09b6f439ea 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/security-authorization.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/security-authorization.md @@ -27,7 +27,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -59,7 +59,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -87,7 +87,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/security-bouncy-castle.md b/site2/website/versioned_docs/version-2.8.3-deprecated/security-bouncy-castle.md index abc25c71b8ae2..be937055d8e31 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/security-bouncy-castle.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/security-bouncy-castle.md @@ -16,7 +16,7 @@ In Pulsar, security and crypto have dependencies on BouncyCastle Jars. For the d `Bouncy Castle` provides both [FIPS](https://www.bouncycastle.org/fips_faq.html) and non-FIPS version. But in a JVM, you can not include both of the 2 versions, and you need to exclude the current version before include the other. -In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. +In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption.md). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. ## How BouncyCastle modules packaged in Pulsar diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/security-extending.md b/site2/website/versioned_docs/version-2.8.3-deprecated/security-extending.md index c088e3a2c1c2c..e7484453b8beb 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/security-extending.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/security-extending.md @@ -9,7 +9,7 @@ Pulsar provides a way to use custom authentication and authorization mechanisms. ## Authentication -Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview). +Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview.md). You can use a custom authentication mechanism by providing the implementation in the form of two plugins. One plugin is for the Client library and the other plugin is for the Pulsar Proxy and/or Pulsar Broker to validate the credentials. diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/security-jwt.md b/site2/website/versioned_docs/version-2.8.3-deprecated/security-jwt.md index 05e641ed67c17..1fa65b7c27f60 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/security-jwt.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/security-jwt.md @@ -32,11 +32,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -305,7 +305,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/security-kerberos.md b/site2/website/versioned_docs/version-2.8.3-deprecated/security-kerberos.md index 670586a9d1691..c49fa3bea1fce 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/security-kerberos.md @@ -373,7 +373,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/security-overview.md b/site2/website/versioned_docs/version-2.8.3-deprecated/security-overview.md index f8e8eac98f805..227dd5b5d4fc3 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/security-overview.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/security-overview.md @@ -21,7 +21,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/security-tls-authentication.md b/site2/website/versioned_docs/version-2.8.3-deprecated/security-tls-authentication.md index f3c32a05d9751..85d2240f41306 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/security-tls-authentication.md @@ -7,9 +7,9 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -103,7 +103,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -123,7 +123,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/security-tls-keystore.md b/site2/website/versioned_docs/version-2.8.3-deprecated/security-tls-keystore.md index 05b919cc15932..2d80782ba88d3 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/security-tls-keystore.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/security-tls-keystore.md @@ -7,7 +7,7 @@ original_id: security-tls-keystore ## Overview -Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication) between clients and Apache Pulsar service. +Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication.md) between clients and Apache Pulsar service. By default it uses PEM format file configuration. This page tries to describe use [KeyStore](https://en.wikipedia.org/wiki/Java_KeyStore) type configure for TLS. @@ -185,7 +185,7 @@ This is similar to [TLS encryption configuing for client with PEM type](security For a a minimal configuration, user need to provide the TrustStore information. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties @@ -278,7 +278,7 @@ webSocketServiceEnabled=false Besides the TLS encryption configuring. The main work is configuring the KeyStore, which contains a valid CN as client role, for client. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/security-tls-transport.md b/site2/website/versioned_docs/version-2.8.3-deprecated/security-tls-transport.md index 92e8dc3c3d280..2cad17a78c350 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,9 +27,9 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). -`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ## Create TLS certificates @@ -130,7 +130,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Configure broker -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -202,7 +202,7 @@ The examples below show that hostname verification is disabled for the CLI tools ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/security-token-admin.md b/site2/website/versioned_docs/version-2.8.3-deprecated/security-token-admin.md index 1a438dc7923ee..a265f6320d28f 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/security-token-admin.md @@ -163,7 +163,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.8.3-deprecated/sql-deployment-configurations.md index 67f7ab17c77d0..02d8bc78f6cb9 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/sql-deployment-configurations.md @@ -64,7 +64,7 @@ Since Pulsar SQL is powered by [Trino (formerly Presto SQL)](https://trino.io), :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/sql-overview.md b/site2/website/versioned_docs/version-2.8.3-deprecated/sql-overview.md index 4a4d5f01b0bc5..0235d0256c5f1 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/sql-overview.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/sql-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Overview" original_id: sql-overview --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL)](https://trino.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL.md)](https://trino.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/standalone.md b/site2/website/versioned_docs/version-2.8.3-deprecated/standalone.md index d114c017d7c00..25afa11a91b11 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/standalone.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > #### Pulsar in production? -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -65,7 +65,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -74,7 +74,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -123,7 +123,7 @@ pulsar-io-aerospike-@pulsar:version@.nar * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -137,7 +137,7 @@ To enable tiered storage feature, follow the instructions below; otherwise skip ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -170,12 +170,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -208,7 +208,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.8.3-deprecated/tiered-storage-overview.md b/site2/website/versioned_docs/version-2.8.3-deprecated/tiered-storage-overview.md index 7c6528ce2b36c..c635034f463b4 100644 --- a/site2/website/versioned_docs/version-2.8.3-deprecated/tiered-storage-overview.md +++ b/site2/website/versioned_docs/version-2.8.3-deprecated/tiered-storage-overview.md @@ -13,9 +13,9 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws). + - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws.md). - - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs). + - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs.md). ::: @@ -25,7 +25,7 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem). + For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem.md). ::: diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/admin-api-brokers.md b/site2/website/versioned_docs/version-2.9.0-deprecated/admin-api-brokers.md index 22409e6c23893..930fe69ecfb0e 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/admin-api-brokers.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/admin-api-brokers.md @@ -176,7 +176,7 @@ One way to configure a Pulsar [broker](reference-terminology.md#broker) is to su But since all broker configuration in Pulsar is stored in ZooKeeper, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values. -* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. +* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. * In the Pulsar admin {@inject: rest:REST:/} API, dynamic configuration is managed through the `/admin/v2/brokers/configuration` endpoint. ### Update dynamic configuration diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/admin-api-clusters.md b/site2/website/versioned_docs/version-2.9.0-deprecated/admin-api-clusters.md index b2f0fa3b25670..1d0c5dc9786f5 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/admin-api-clusters.md @@ -89,7 +89,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -113,11 +113,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block > - For the latest and complete information about `Java admin API`, including classes, methods, descriptions, and more, see [Java admin API doc](https://pulsar.apache.org/api/admin/). -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -118,7 +118,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-cgo.md b/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-cgo.md index c79f7bb965473..f352f942b7714 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-cgo.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-cgo.md @@ -24,7 +24,7 @@ Currently, the following Go clients are maintained in two repositories. ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Install go package @@ -57,7 +57,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -75,7 +75,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -236,14 +236,14 @@ Parameter | Description | Default `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | `Properties`| Attach a set of application defined properties to the producer. This properties will be visible in the topic stats | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | Set the number of max pending messages across all the partitions. This setting will be used to lower the max pending messages for each partition `MaxPendingMessages(int)`, if the total exceeds the configured value.| `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | `Batching` | Control whether automatic batching of messages is enabled for the producer. | false `BatchingMaxPublishDelay` | Set the time period within which the messages sent will be batched (default: 1ms) if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or until | 1ms `BatchingMaxMessages` | Set the maximum number of messages permitted in a batch. (default: 1000) If set to a value greater than 1, messages will be queued until this threshold is reached or batch interval has elapsed | 1000 @@ -512,7 +512,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-cpp.md index 3c94293b7929c..455cf02116d50 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-cpp.md @@ -602,7 +602,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema This section describes some examples about schema. For more information about -schema, see [Pulsar schema](schema-get-started). +schema, see [Pulsar schema](schema-get-started.md). ### Avro schema diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-dotnet.md b/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-dotnet.md index 4e0afe3d2ff6b..fbec5e473be69 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-dotnet.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-dotnet.md @@ -170,7 +170,7 @@ var client = PulsarClient.Builder() Currently, the Pulsar C# client supports the TLS (Transport Layer Security) and JWT (JSON Web Token) authentication. -If you have followed [Authentication using TLS](security-tls-authentication), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: +If you have followed [Authentication using TLS](security-tls-authentication.md), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: 1. Create an unencrypted and password-less pfx file. diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-go.md b/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-go.md index fbb2573aa5db8..6281b03dd8c80 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-go.md @@ -35,7 +35,7 @@ import "github.com/apache/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -61,7 +61,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -844,7 +844,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker @@ -864,7 +864,7 @@ opts := pulsar.ClientOptions{ ## OAuth2 authentication -To use [OAuth2 authentication](security-oauth2), you'll need to configure your client to perform the following operations. +To use [OAuth2 authentication](security-oauth2.md), you'll need to configure your client to perform the following operations. This example shows how to configure OAuth2 authentication. ```go diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-java.md b/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-java.md index e87c5e6c23c8a..c3d41a3f13da2 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-java.md @@ -5,7 +5,7 @@ sidebar_label: "Java" original_id: client-libraries-java --- -You can use a Pulsar Java client to create the Java [producer](#producer), [consumer](#consumer), and [readers](#reader) of messages and to perform [administrative tasks](admin-api-overview). The current Java client version is **@pulsar:version@**. +You can use a Pulsar Java client to create the Java [producer](#producer), [consumer](#consumer), and [readers](#reader) of messages and to perform [administrative tasks](admin-api-overview.md). The current Java client version is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), and [reader](#reader) of a Java client are thread-safe. @@ -14,10 +14,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Include both `pulsar-client` and `pulsar-client-admin`
    Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.
    In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -57,7 +57,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -83,7 +83,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -114,7 +114,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -240,7 +240,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics cookbook](cookbooks-partitioned). +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics cookbook](cookbooks-partitioned.md). ### Async send @@ -827,7 +827,7 @@ Total hash range size is 65536, so the max end of the range should be less than ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producer) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producer) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -945,7 +945,7 @@ For example of ProtobufNativeSchema, see [`SchemaDefinition` in `Complex type`]( ## Authentication -Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2). You can use the Pulsar Java client with all of them. +Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2.md). You can use the Pulsar Java client with all of them. ### TLS Authentication @@ -972,7 +972,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` @@ -1009,7 +1009,7 @@ PulsarClient client = PulsarClient.builder() ### Oauth2 -The following example shows how to use [Oauth2](security-oauth2) as an authentication provider for the Pulsar Java client. +The following example shows how to use [Oauth2](security-oauth2.md) as an authentication provider for the Pulsar Java client. You can use the factory method to configure authentication for Pulsar Java client. diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-node.md b/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-node.md index e6584ede93f6d..1ff37b2629466 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-node.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-node.md @@ -48,7 +48,7 @@ Also, this library works only in Node.js 10.x or later because it uses the [`nod ::: ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here is an example for `localhost`: @@ -66,7 +66,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication), the URL looks like this: +If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication.md), the URL looks like this: ```http @@ -101,7 +101,7 @@ The following configurable parameters are available for Pulsar clients: | Parameter | Description | Default | | :-------- | :---------- | :------ | | `serviceUrl` | The connection URL for the Pulsar cluster. See [above](#connection-urls) for more info. | | -| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication) for more info. | | +| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication.md) for more info. | | | `operationTimeoutSeconds` | The timeout for Node.js client operations (creating producers, subscribing to and unsubscribing from [topics](reference-terminology.md#topic)). Retries occur until this threshold is reached, at which point the operation fails. | 30 | | `ioThreads` | The number of threads to use for handling connections to Pulsar [brokers](reference-terminology.md#broker). | 1 | | `messageListenerThreads` | The number of threads used by message listeners ([consumers](#consumers) and [readers](#readers)). | 1 | @@ -154,7 +154,7 @@ Pulsar Node.js producers have the following methods available: | :-------- | :---------- | :------ | | `topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer publishes messages. | | | `producerName` | A name for the producer. If you do not explicitly assign a name, Pulsar automatically generates a globally unique name. If you choose to explicitly assign a name, it needs to be unique across *all* Pulsar clusters, otherwise the creation operation throws an error. | | -| `sendTimeoutMs` | When publishing a message to a topic, the producer waits for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error is thrown. If you set `sendTimeoutMs` to -1, the timeout is set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30000 | +| `sendTimeoutMs` | When publishing a message to a topic, the producer waits for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error is thrown. If you set `sendTimeoutMs` to -1, the timeout is set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30000 | | `initialSequenceId` | The initial sequence ID of the message. When producer send message, add sequence ID to message. The ID is increased each time to send. | | | `maxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `send` method fails *unless* `blockIfQueueFull` is set to `true`. | 1000 | | `maxPendingMessagesAcrossPartitions` | The maximum size of the sum of partition's pending queue. | 50000 | diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-python.md b/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-python.md index 72aaf211ae2b2..90cc840daa0a8 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [Python directory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [Python directory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-websocket.md index c89f88365defb..60970c7ea4df2 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/client-libraries-websocket.md @@ -5,14 +5,14 @@ sidebar_label: "WebSocket" original_id: client-libraries-websocket --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-architecture-overview.md index 95cf02eac109c..4baa8c30a0d00 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-architecture-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Architecture" original_id: concepts-architecture-overview --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -17,20 +17,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers. The producers connect to the brokers to publish messages and the consumers connect to the brokers to consume the messages. -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -40,9 +40,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -136,17 +136,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -154,7 +154,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-authentication.md b/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-authentication.md index b375ecb16a2d8..f6307890c904a 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-authentication.md @@ -5,5 +5,5 @@ sidebar_label: "Authentication and Authorization" original_id: concepts-authentication --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-clients.md b/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-clients.md index b68f76a2d8b08..4040624f7d636 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-clients.md @@ -5,12 +5,12 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -38,7 +38,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-messaging.md b/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-messaging.md index a0f7d046badc7..dbe540c5df8e2 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-messaging.md @@ -23,8 +23,8 @@ Messages are the basic "unit" of Pulsar. The following table lists the component Component | Description :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer who produces the message. If you do not specify a producer name, the default name is used. Sequence ID | Each Pulsar message belongs to an ordered sequence on its topic. The sequence ID of the message is its order in that sequence. @@ -52,7 +52,7 @@ The default size of a message is 5 MB. You can configure the max size of a messa ``` -> For more information on Pulsar messages, see Pulsar [binary protocol](developing-binary-protocol). +> For more information on Pulsar messages, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -149,7 +149,7 @@ Messages are received from [brokers](reference-terminology.md#broker) either syn ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -502,7 +502,7 @@ Decisions about routing and subscription modes can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription modes work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -514,7 +514,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -552,7 +552,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -639,7 +639,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -662,12 +662,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication.md). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-multi-tenancy.md index 730458c904618..93a59557b2efc 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-multi-tenancy.md @@ -5,7 +5,7 @@ sidebar_label: "Multi Tenancy" original_id: concepts-multi-tenancy --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -21,7 +21,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -29,7 +29,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-overview.md b/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-overview.md index 310a78dff4565..c643aa0ce7bbc 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription modes](concepts-messaging.md#subscription-modes) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-replication.md b/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-replication.md index 6e23962e99771..799f0eb4d92c6 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-replication.md @@ -5,5 +5,5 @@ sidebar_label: "Geo Replication" original_id: concepts-replication --- -Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that. +Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo.md) in Pulsar enables you to do that. diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-tiered-storage.md index 0b45b0a2ed501..f6988e53a8cd4 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-topic-compaction.md index e402c965384fd..34b7ed7fbbd31 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/concepts-topic-compaction.md @@ -7,7 +7,7 @@ original_id: concepts-topic-compaction Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -24,7 +24,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.9.0-deprecated/cookbooks-compaction.md index 6fcbc197f9ecb..dfa314727241a 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.9.0-deprecated/cookbooks-non-persistent.md index 391569a8cbc6b..178301e86eb8d 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/cookbooks-non-persistent.md @@ -41,7 +41,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -55,7 +55,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.9.0-deprecated/cookbooks-partitioned.md index 7882fb9025312..fb9ac354cc6d6 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/cookbooks-partitioned.md @@ -4,4 +4,4 @@ title: Partitioned topics sidebar_label: "Partitioned Topics" original_id: cookbooks-partitioned --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.9.0-deprecated/cookbooks-retention-expiry.md index 0283a64312ca7..c8c46b3caa1be 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/cookbooks-retention-expiry.md @@ -22,7 +22,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-aws.md b/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-aws.md index 071effeffde24..93c389b56e2cf 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-aws.md @@ -5,7 +5,7 @@ sidebar_label: "Amazon Web Services" original_id: deploy-aws --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -210,7 +210,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors) , edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md) , edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-bare-metal-multi-cluster.md index df25604af4e62..f25b11041c5e3 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-bare-metal-multi-cluster.md @@ -7,13 +7,13 @@ original_id: deploy-bare-metal-multi-cluster :::tip -1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster). -2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster.md). +2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo).Deploying a multi-cluster Pulsar instance consists of the following steps: +A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md).Deploying a multi-cluster Pulsar instance consists of the following steps: 1. Deploying two separate ZooKeeper quorums: a local quorum for each cluster in the instance and a configuration store quorum for instance-wide tasks 2. Initializing cluster metadata for each cluster @@ -22,7 +22,7 @@ A Pulsar instance consists of multiple Pulsar clusters working in unison. You ca > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. ## System requirement @@ -65,7 +65,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -244,7 +244,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -363,16 +363,16 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. -You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > **Service discovery already provided by many scheduling systems** -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-bare-metal.md index ed0c9d7da465b..9bb4235cece5d 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-bare-metal.md @@ -7,9 +7,9 @@ original_id: deploy-bare-metal :::tip -1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster). -2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster.md). +2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -126,7 +126,7 @@ The extracted directory contains the following subdirectories: Directory | Contains :---------|:-------- -`bin` |[command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` |[command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `data` | The data storage directory that ZooKeeper and BookKeeper use `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -279,9 +279,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (you had better not use a different port). -`--web-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). +`--web-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should not use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (you had better not use a different port). -`--broker-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). +`--broker-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). > If you do not have a DNS server, you can use multi-host format in the service URL with the following settings: @@ -421,7 +421,7 @@ webServicePortTls=8443 ### Enable Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -439,7 +439,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker). +If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker.md). ### Start Brokers diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-dcos.md b/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-dcos.md index 49cf624ea24f1..35a0a83d716ad 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -To enable all built-in [Pulsar IO](io-overview) connectors in your Pulsar deployment, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). +To enable all built-in [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-docker.md b/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-docker.md index 64a0939aa787b..8348d78deb237 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-docker.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-docker.md @@ -57,4 +57,4 @@ docker network connect pulsar broker To check whether the containers are successfully connected to the network, enter the `docker network inspect pulsar` command. -For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal). +For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal.md). diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-kubernetes.md index dc7123d000467..1aefc6ad79f71 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-kubernetes.md @@ -6,6 +6,6 @@ original_id: deploy-kubernetes --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-monitoring.md b/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-monitoring.md index 87d435050df27..2b5c19344dc8c 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/deploy-monitoring.md @@ -113,7 +113,7 @@ http://$FUNCTIONS_WORKER_ADDRESS:$WORKER_PORT/metrics: You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy). +When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy.md). ## Dashboards @@ -121,7 +121,7 @@ When you collect time series statistics, the major problem is to make sure the n ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager). +The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/develop-schema.md b/site2/website/versioned_docs/version-2.9.0-deprecated/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/develop-schema.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/functions-deploy.md b/site2/website/versioned_docs/version-2.9.0-deprecated/functions-deploy.md index d8fe0fc14e5bb..2a0d68d6c623c 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/functions-deploy.md @@ -9,12 +9,12 @@ original_id: functions-deploy To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface @@ -179,7 +179,7 @@ $ bin/pulsar-admin functions create \ Package management enables version management and simplifies the upgrade and rollback processes for Functions, Sinks, and Sources. When you use the same function, sink and source in different namespaces, you can upload them to a common package management system. -To use [Package management service](admin-api-packages), ensure that the package management service has been enabled in your cluster by setting the following properties in `broker.conf`. +To use [Package management service](admin-api-packages.md), ensure that the package management service has been enabled in your cluster by setting the following properties in `broker.conf`. > Note: Package management service is not enabled by default. diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/functions-develop.md b/site2/website/versioned_docs/version-2.9.0-deprecated/functions-develop.md index 36ea4b4a21073..2e29aa1c47400 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/functions-develop.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/functions-develop.md @@ -1144,7 +1144,7 @@ You can monitor Pulsar Functions that have been deployed with the following meth - Check the metrics provided by Pulsar. - Pulsar Functions expose the metrics that can be collected and used for monitoring the health of **Java, Python, and Go** functions. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar Functions expose the metrics that can be collected and used for monitoring the health of **Java, Python, and Go** functions. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. For the complete list of the function metrics, see [here](reference-metrics.md#pulsar-functions). @@ -1214,7 +1214,7 @@ Currently, the feature is not available in Go. ## Security -If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker). For more details, refer to [Security settings](functions-worker.md#security-settings). +If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings). Pulsar Functions can support the following providers: diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/functions-overview.md b/site2/website/versioned_docs/version-2.9.0-deprecated/functions-overview.md index 25dc6029acf70..816d301e0fd0e 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/functions-overview.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/functions-overview.md @@ -164,7 +164,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/functions-package.md b/site2/website/versioned_docs/version-2.9.0-deprecated/functions-package.md index 2666d5944b2e0..db2c4e987dc7b 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/functions-package.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/functions-package.md @@ -15,7 +15,7 @@ Currently, the window function is not available in Python and Go. ## Prerequisite -Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm). +Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm.md). To check whether the Docker image starts, you can use the `docker ps` command. diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/functions-worker.md b/site2/website/versioned_docs/version-2.9.0-deprecated/functions-worker.md index 78f4f19487951..49fc76b30bdaa 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/functions-worker.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/functions-worker.md @@ -4,7 +4,7 @@ title: Deploy and manage functions worker sidebar_label: "Setup: Pulsar Functions Worker" original_id: functions-worker --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -179,7 +179,7 @@ brokerClientTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -199,7 +199,7 @@ authenticationProviders: [ provider1, provider2 ] ``` For *TLS Authentication* provider, follow the example below to add the necessary settings. -See [TLS Authentication](security-tls-authentication) for more details. +See [TLS Authentication](security-tls-authentication.md) for more details. ``` @@ -223,7 +223,7 @@ properties: ``` For *Token Authentication* provider, add necessary settings for `properties` if needed. -See [Token Authentication](security-jwt) for more details. +See [Token Authentication](security-jwt.md) for more details. Note: key files must be DER-encoded ``` @@ -262,7 +262,7 @@ superUserRoles: You can use the public and private key pair that the application configures to perform encryption. Only the consumers with a valid key can decrypt the encrypted messages. -To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption). +To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption.md). We include the relevant configuration information of `CryptoConfig` into `ProducerConfig`. The specific configurable field information about `CryptoConfig` is as follows: diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/getting-started-docker.md b/site2/website/versioned_docs/version-2.9.0-deprecated/getting-started-docker.md index 71f1bb9734fb7..de5ead69e164b 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/getting-started-docker.md @@ -43,8 +43,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -103,7 +103,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/getting-started-helm.md b/site2/website/versioned_docs/version-2.9.0-deprecated/getting-started-helm.md index 98a3097d436db..5e9f7044a6d74 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/getting-started-helm.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/getting-started-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -409,7 +409,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/getting-started-pulsar.md b/site2/website/versioned_docs/version-2.9.0-deprecated/getting-started-pulsar.md index 11c5e66cb62c2..752590f57b558 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/getting-started-pulsar.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/getting-started-pulsar.md @@ -5,13 +5,13 @@ sidebar_label: "Pulsar 2.0" original_id: pulsar-2.0 --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/getting-started-standalone.md b/site2/website/versioned_docs/version-2.9.0-deprecated/getting-started-standalone.md index 436355aa832bc..9137ba291421a 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > **Pulsar in production?** -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -65,7 +65,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -74,7 +74,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -122,7 +122,7 @@ pulsar-io-aerospike-@pulsar:version@.nar :::note * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -135,7 +135,7 @@ pulsar-io-aerospike-@pulsar:version@.nar ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -168,12 +168,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -205,7 +205,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/helm-install.md b/site2/website/versioned_docs/version-2.9.0-deprecated/helm-install.md index 80fb7f8d6c8d6..9f81f52e0dab1 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/helm-install.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/helm-install.md @@ -21,7 +21,7 @@ Before deploying Pulsar, you need to prepare your environment. ### Tools -Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools) on your computer. +Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools.md) on your computer. ## Cloud cluster preparation @@ -31,8 +31,8 @@ To create and connect to the Kubernetes cluster, follow the instructions: ## Pulsar deployment -Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy). +Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy.md). ## Pulsar upgrade -To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade). +To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade.md). diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/helm-overview.md b/site2/website/versioned_docs/version-2.9.0-deprecated/helm-overview.md index fc4cbfe426a93..125f595cbe68a 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/helm-overview.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/helm-overview.md @@ -52,9 +52,9 @@ Moreover, Helm chart supports: ## Quick start -To run with Apache Pulsar Helm chart as fast as possible in a **non-production** use case, we provide a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +To run with Apache Pulsar Helm chart as fast as possible in a **non-production** use case, we provide a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -This guide walks you through deploying Apache Pulsar Helm chart with default values and features, but it is *not* suitable for deployments in production-ready environments. To deploy the charts in production under sustained load, you can follow the complete [Installation Guide](helm-install). +This guide walks you through deploying Apache Pulsar Helm chart with default values and features, but it is *not* suitable for deployments in production-ready environments. To deploy the charts in production under sustained load, you can follow the complete [Installation Guide](helm-install.md). ## Troubleshooting @@ -64,7 +64,7 @@ Although we have done our best to make these charts as seamless as possible, tro The Apache Pulsar Helm chart contains all required dependencies. -If you deploy a PoC for testing, we strongly suggest you follow this [Quick Start Guide](getting-started-helm) for your first iteration. +If you deploy a PoC for testing, we strongly suggest you follow this [Quick Start Guide](getting-started-helm.md) for your first iteration. 1. [Preparation](helm-prepare.md) 2. [Deployment](helm-deploy.md) @@ -82,7 +82,7 @@ helm upgrade apache/pulsar -f pulsar.yaml ``` -For more detailed information, see [Upgrading](helm-upgrade). +For more detailed information, see [Upgrading](helm-upgrade.md). ## Uninstallation diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/helm-tools.md b/site2/website/versioned_docs/version-2.9.0-deprecated/helm-tools.md index efd61eae32506..6ba89006913b6 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/helm-tools.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/helm-tools.md @@ -25,7 +25,7 @@ You can get Helm from the project's [releases page](https://github.com/helm/helm ### Next steps -Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare). +Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare.md). ## Additional information diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/helm-upgrade.md b/site2/website/versioned_docs/version-2.9.0-deprecated/helm-upgrade.md index e39cf97407355..7d671e6bfb3c1 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/helm-upgrade.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/helm-upgrade.md @@ -20,7 +20,7 @@ You can retrieve your previous `--set` arguments cleanly, with `helm get values To upgrade Apache Pulsar to a newer version, follow these steps: 1. Check the change log for the specific version you would like to upgrade to. -2. Go through [deployment documentation](helm-deploy) step by step. +2. Go through [deployment documentation](helm-deploy.md) step by step. 3. Extract your previous `--set` arguments with the following command. ```bash diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/io-cdc.md b/site2/website/versioned_docs/version-2.9.0-deprecated/io-cdc.md index 20f16ae0d6904..e6e662884826d 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/io-cdc.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/io-cdc.md @@ -13,8 +13,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/io-develop.md b/site2/website/versioned_docs/version-2.9.0-deprecated/io-develop.md index a20206fbf6a06..d6f4f8261ac82 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/io-develop.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/io-develop.md @@ -14,15 +14,15 @@ import TabItem from '@theme/TabItem'; This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -275,7 +275,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -304,7 +304,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). @@ -379,7 +379,7 @@ Pulsar connectors enable you to move data in and out of Pulsar easily. It is imp - Check the metrics provided by Pulsar. - Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. - Set and check your customized metrics. diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/io-overview.md b/site2/website/versioned_docs/version-2.9.0-deprecated/io-overview.md index 0070a015296db..3db5ee34042d3 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/io-overview.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/io-overview.md @@ -158,7 +158,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/io-quickstart.md b/site2/website/versioned_docs/version-2.9.0-deprecated/io-quickstart.md index 00bcba3769ff7..40eaf5c1de221 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you are able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you are able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However, all the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -109,7 +109,7 @@ This section demonstrates how to connect Pulsar to Cassandra. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink.md). ::: @@ -236,11 +236,11 @@ You can create a configuration file through one of the following methods. ``` -For more information, see [Cassandra sink connector](io-cassandra-sink). +For more information, see [Cassandra sink connector](io-cassandra-sink.md). ### Create a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on them. Run the following command to create a Cassandra sink connector with sink type _cassandra_ and the config file _examples/cassandra-sink.yml_ created previously. @@ -267,7 +267,7 @@ as a Pulsar Function and writes the messages produced in the topic _test_cassand ### Inspect a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * Get the information of a Cassandra sink. @@ -431,7 +431,7 @@ to monitor a connector and perform other operations on it. ### Delete a Cassandra Sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. ```bash @@ -454,7 +454,7 @@ This section demonstrates how to connect Pulsar to PostgreSQL. ::: ->For more information, see [JDBC sink connector](io-jdbc-sink). +>For more information, see [JDBC sink connector](io-jdbc-sink.md). ### Setup a PostgreSQL cluster @@ -614,7 +614,7 @@ In this section, you need to configure a JDBC sink connector. ### Create a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on it. This example creates a sink connector and specifies the desired information. @@ -660,7 +660,7 @@ The sink has been created successfully if the following message appears. ### Inspect a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * List all running JDBC sink(s). @@ -780,7 +780,7 @@ to monitor a connector and perform other operations on it. ### Stop a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to stop a connector and perform other operations on it. ```bash @@ -808,7 +808,7 @@ The sink instance has been stopped successfully if the following message disappe ### Restart a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to restart a connector and perform other operations on it. ```bash @@ -844,7 +844,7 @@ Note that `pulsar-admin sinks localrun options` **runs a sink connector locally* ### Update a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to update a connector and perform other operations on it. This example updates the parallelism of the _pulsar-postgres-jdbc-sink_ sink connector to 2. @@ -912,7 +912,7 @@ The result shows that the parallelism is 2. ### Delete a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. This example deletes the _pulsar-postgres-jdbc-sink_ sink connector. diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/io-use.md b/site2/website/versioned_docs/version-2.9.0-deprecated/io-use.md index b6409d4420ad7..da9ed746c4d37 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/io-use.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/io-use.md @@ -15,7 +15,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/kubernetes-helm.md b/site2/website/versioned_docs/version-2.9.0-deprecated/kubernetes-helm.md index a9f768e6a96b3..ea92a0968cd7d 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/kubernetes-helm.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/kubernetes-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/reference-cli-tools.md b/site2/website/versioned_docs/version-2.9.0-deprecated/reference-cli-tools.md index eff2625277504..6893da3ec6394 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/reference-configuration.md b/site2/website/versioned_docs/version-2.9.0-deprecated/reference-configuration.md index e6b8bbf982c0d..2cf706608298e 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/reference-configuration.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/reference-configuration.md @@ -8,7 +8,7 @@ original_id: reference-configuration -You can manage Pulsar configuration by configuration files in the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a Pulsar [installation](getting-started-standalone). +You can manage Pulsar configuration by configuration files in the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a Pulsar [installation](getting-started-standalone.md). - [BookKeeper](#bookkeeper) - [Broker](#broker) diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/schema-get-started.md b/site2/website/versioned_docs/version-2.9.0-deprecated/schema-get-started.md index 97e270586a26d..afacb0fa51f2e 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/schema-get-started.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/schema-get-started.md @@ -17,7 +17,7 @@ Applications typically adopt one of the following approaches to guarantee type s #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ### Client-side approach diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/schema-manage.md b/site2/website/versioned_docs/version-2.9.0-deprecated/schema-manage.md index 4ae363882b2d5..c588aae619eee 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/schema-manage.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/schema-manage.md @@ -160,7 +160,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/schema-understand.md b/site2/website/versioned_docs/version-2.9.0-deprecated/schema-understand.md index f454dd6873e8e..55bc662c66633 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/schema-understand.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/schema-understand.md @@ -479,7 +479,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```java diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/security-athenz.md b/site2/website/versioned_docs/version-2.9.0-deprecated/security-athenz.md index 947c3f470be46..8a39fe25316d0 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/security-athenz.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/security-athenz.md @@ -76,7 +76,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/security-authorization.md b/site2/website/versioned_docs/version-2.9.0-deprecated/security-authorization.md index cc9c03d5fc1dd..9cfd7c8c203f6 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/security-authorization.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/security-authorization.md @@ -27,7 +27,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -59,7 +59,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -87,7 +87,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication @@ -117,7 +117,7 @@ PulsarAdmin admin = PulsarAdmin.builder() When a client is identified with multiple roles in a token (the type of role claim in the token is an array) during the authentication process, Pulsar supports to check the permissions of all the roles and further authorize the client as long as one of its roles has the required permissions. > **Note**
    -> This authorization method is only compatible with [JWT authentication](security-jwt). +> This authorization method is only compatible with [JWT authentication](security-jwt.md). To enable this authorization method, configure the authorization provider as `MultiRolesTokenAuthorizationProvider` in the `conf/broker.conf` file. diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/security-bouncy-castle.md b/site2/website/versioned_docs/version-2.9.0-deprecated/security-bouncy-castle.md index abc25c71b8ae2..be937055d8e31 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/security-bouncy-castle.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/security-bouncy-castle.md @@ -16,7 +16,7 @@ In Pulsar, security and crypto have dependencies on BouncyCastle Jars. For the d `Bouncy Castle` provides both [FIPS](https://www.bouncycastle.org/fips_faq.html) and non-FIPS version. But in a JVM, you can not include both of the 2 versions, and you need to exclude the current version before include the other. -In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. +In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption.md). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. ## How BouncyCastle modules packaged in Pulsar diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/security-extending.md b/site2/website/versioned_docs/version-2.9.0-deprecated/security-extending.md index c088e3a2c1c2c..e7484453b8beb 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/security-extending.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/security-extending.md @@ -9,7 +9,7 @@ Pulsar provides a way to use custom authentication and authorization mechanisms. ## Authentication -Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview). +Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview.md). You can use a custom authentication mechanism by providing the implementation in the form of two plugins. One plugin is for the Client library and the other plugin is for the Pulsar Proxy and/or Pulsar Broker to validate the credentials. diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/security-jwt.md b/site2/website/versioned_docs/version-2.9.0-deprecated/security-jwt.md index 05e641ed67c17..1fa65b7c27f60 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/security-jwt.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/security-jwt.md @@ -32,11 +32,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -305,7 +305,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/security-kerberos.md b/site2/website/versioned_docs/version-2.9.0-deprecated/security-kerberos.md index 670586a9d1691..c49fa3bea1fce 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/security-kerberos.md @@ -373,7 +373,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/security-overview.md b/site2/website/versioned_docs/version-2.9.0-deprecated/security-overview.md index f8e8eac98f805..227dd5b5d4fc3 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/security-overview.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/security-overview.md @@ -21,7 +21,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/security-tls-authentication.md b/site2/website/versioned_docs/version-2.9.0-deprecated/security-tls-authentication.md index f3c32a05d9751..85d2240f41306 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/security-tls-authentication.md @@ -7,9 +7,9 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -103,7 +103,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -123,7 +123,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/security-tls-keystore.md b/site2/website/versioned_docs/version-2.9.0-deprecated/security-tls-keystore.md index 6cc3a2c60835e..0b3b50fcebb10 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/security-tls-keystore.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/security-tls-keystore.md @@ -7,7 +7,7 @@ original_id: security-tls-keystore ## Overview -Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication) between clients and Apache Pulsar service. +Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication.md) between clients and Apache Pulsar service. By default it uses PEM format file configuration. This page tries to describe use [KeyStore](https://en.wikipedia.org/wiki/Java_KeyStore) type configure for TLS. @@ -185,7 +185,7 @@ This is similar to [TLS encryption configuing for client with PEM type](security For a a minimal configuration, user need to provide the TrustStore information. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties @@ -278,7 +278,7 @@ webSocketServiceEnabled=false Besides the TLS encryption configuring. The main work is configuring the KeyStore, which contains a valid CN as client role, for client. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/security-tls-transport.md b/site2/website/versioned_docs/version-2.9.0-deprecated/security-tls-transport.md index 92e8dc3c3d280..2cad17a78c350 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,9 +27,9 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). -`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ## Create TLS certificates @@ -130,7 +130,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Configure broker -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -202,7 +202,7 @@ The examples below show that hostname verification is disabled for the CLI tools ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/security-token-admin.md b/site2/website/versioned_docs/version-2.9.0-deprecated/security-token-admin.md index 1a438dc7923ee..a265f6320d28f 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/security-token-admin.md @@ -163,7 +163,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.9.0-deprecated/sql-deployment-configurations.md index 67f7ab17c77d0..02d8bc78f6cb9 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/sql-deployment-configurations.md @@ -64,7 +64,7 @@ Since Pulsar SQL is powered by [Trino (formerly Presto SQL)](https://trino.io), :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/sql-overview.md b/site2/website/versioned_docs/version-2.9.0-deprecated/sql-overview.md index 4a4d5f01b0bc5..0235d0256c5f1 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/sql-overview.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/sql-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Overview" original_id: sql-overview --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL)](https://trino.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL.md)](https://trino.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/standalone-docker.md b/site2/website/versioned_docs/version-2.9.0-deprecated/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/standalone.md b/site2/website/versioned_docs/version-2.9.0-deprecated/standalone.md index 32970bfcfe24f..9bff74d92c10b 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/standalone.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > **Pulsar in production?** -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -65,7 +65,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -74,7 +74,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -122,7 +122,7 @@ pulsar-io-aerospike-@pulsar:version@.nar :::note * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -135,7 +135,7 @@ pulsar-io-aerospike-@pulsar:version@.nar ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -168,12 +168,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -205,7 +205,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.9.0-deprecated/tiered-storage-overview.md b/site2/website/versioned_docs/version-2.9.0-deprecated/tiered-storage-overview.md index 7c6528ce2b36c..c635034f463b4 100644 --- a/site2/website/versioned_docs/version-2.9.0-deprecated/tiered-storage-overview.md +++ b/site2/website/versioned_docs/version-2.9.0-deprecated/tiered-storage-overview.md @@ -13,9 +13,9 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws). + - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws.md). - - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs). + - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs.md). ::: @@ -25,7 +25,7 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem). + For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem.md). ::: diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/admin-api-brokers.md b/site2/website/versioned_docs/version-2.9.1-deprecated/admin-api-brokers.md index 22409e6c23893..930fe69ecfb0e 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/admin-api-brokers.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/admin-api-brokers.md @@ -176,7 +176,7 @@ One way to configure a Pulsar [broker](reference-terminology.md#broker) is to su But since all broker configuration in Pulsar is stored in ZooKeeper, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values. -* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. +* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. * In the Pulsar admin {@inject: rest:REST:/} API, dynamic configuration is managed through the `/admin/v2/brokers/configuration` endpoint. ### Update dynamic configuration diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/admin-api-clusters.md b/site2/website/versioned_docs/version-2.9.1-deprecated/admin-api-clusters.md index b2f0fa3b25670..1d0c5dc9786f5 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/admin-api-clusters.md @@ -89,7 +89,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -113,11 +113,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block > - For the latest and complete information about `Java admin API`, including classes, methods, descriptions, and more, see [Java admin API doc](https://pulsar.apache.org/api/admin/). -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -118,7 +118,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-cgo.md b/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-cgo.md index c79f7bb965473..f352f942b7714 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-cgo.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-cgo.md @@ -24,7 +24,7 @@ Currently, the following Go clients are maintained in two repositories. ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Install go package @@ -57,7 +57,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -75,7 +75,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -236,14 +236,14 @@ Parameter | Description | Default `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | `Properties`| Attach a set of application defined properties to the producer. This properties will be visible in the topic stats | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | Set the number of max pending messages across all the partitions. This setting will be used to lower the max pending messages for each partition `MaxPendingMessages(int)`, if the total exceeds the configured value.| `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | `Batching` | Control whether automatic batching of messages is enabled for the producer. | false `BatchingMaxPublishDelay` | Set the time period within which the messages sent will be batched (default: 1ms) if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or until | 1ms `BatchingMaxMessages` | Set the maximum number of messages permitted in a batch. (default: 1000) If set to a value greater than 1, messages will be queued until this threshold is reached or batch interval has elapsed | 1000 @@ -512,7 +512,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-cpp.md index 3c94293b7929c..455cf02116d50 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-cpp.md @@ -602,7 +602,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema This section describes some examples about schema. For more information about -schema, see [Pulsar schema](schema-get-started). +schema, see [Pulsar schema](schema-get-started.md). ### Avro schema diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-dotnet.md b/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-dotnet.md index ade664cd95277..b574fa0b2e5ed 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-dotnet.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-dotnet.md @@ -170,7 +170,7 @@ var client = PulsarClient.Builder() Currently, the Pulsar C# client supports the TLS (Transport Layer Security) and JWT (JSON Web Token) authentication. -If you have followed [Authentication using TLS](security-tls-authentication), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: +If you have followed [Authentication using TLS](security-tls-authentication.md), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: 1. Create an unencrypted and password-less pfx file. diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-go.md b/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-go.md index fbb2573aa5db8..6281b03dd8c80 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-go.md @@ -35,7 +35,7 @@ import "github.com/apache/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -61,7 +61,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -844,7 +844,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker @@ -864,7 +864,7 @@ opts := pulsar.ClientOptions{ ## OAuth2 authentication -To use [OAuth2 authentication](security-oauth2), you'll need to configure your client to perform the following operations. +To use [OAuth2 authentication](security-oauth2.md), you'll need to configure your client to perform the following operations. This example shows how to configure OAuth2 authentication. ```go diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-java.md b/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-java.md index e87c5e6c23c8a..c3d41a3f13da2 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-java.md @@ -5,7 +5,7 @@ sidebar_label: "Java" original_id: client-libraries-java --- -You can use a Pulsar Java client to create the Java [producer](#producer), [consumer](#consumer), and [readers](#reader) of messages and to perform [administrative tasks](admin-api-overview). The current Java client version is **@pulsar:version@**. +You can use a Pulsar Java client to create the Java [producer](#producer), [consumer](#consumer), and [readers](#reader) of messages and to perform [administrative tasks](admin-api-overview.md). The current Java client version is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), and [reader](#reader) of a Java client are thread-safe. @@ -14,10 +14,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Include both `pulsar-client` and `pulsar-client-admin`
    Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.
    In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -57,7 +57,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -83,7 +83,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -114,7 +114,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -240,7 +240,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics cookbook](cookbooks-partitioned). +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics cookbook](cookbooks-partitioned.md). ### Async send @@ -827,7 +827,7 @@ Total hash range size is 65536, so the max end of the range should be less than ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producer) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producer) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -945,7 +945,7 @@ For example of ProtobufNativeSchema, see [`SchemaDefinition` in `Complex type`]( ## Authentication -Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2). You can use the Pulsar Java client with all of them. +Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2.md). You can use the Pulsar Java client with all of them. ### TLS Authentication @@ -972,7 +972,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` @@ -1009,7 +1009,7 @@ PulsarClient client = PulsarClient.builder() ### Oauth2 -The following example shows how to use [Oauth2](security-oauth2) as an authentication provider for the Pulsar Java client. +The following example shows how to use [Oauth2](security-oauth2.md) as an authentication provider for the Pulsar Java client. You can use the factory method to configure authentication for Pulsar Java client. diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-node.md b/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-node.md index e6584ede93f6d..1ff37b2629466 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-node.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-node.md @@ -48,7 +48,7 @@ Also, this library works only in Node.js 10.x or later because it uses the [`nod ::: ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here is an example for `localhost`: @@ -66,7 +66,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication), the URL looks like this: +If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication.md), the URL looks like this: ```http @@ -101,7 +101,7 @@ The following configurable parameters are available for Pulsar clients: | Parameter | Description | Default | | :-------- | :---------- | :------ | | `serviceUrl` | The connection URL for the Pulsar cluster. See [above](#connection-urls) for more info. | | -| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication) for more info. | | +| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication.md) for more info. | | | `operationTimeoutSeconds` | The timeout for Node.js client operations (creating producers, subscribing to and unsubscribing from [topics](reference-terminology.md#topic)). Retries occur until this threshold is reached, at which point the operation fails. | 30 | | `ioThreads` | The number of threads to use for handling connections to Pulsar [brokers](reference-terminology.md#broker). | 1 | | `messageListenerThreads` | The number of threads used by message listeners ([consumers](#consumers) and [readers](#readers)). | 1 | @@ -154,7 +154,7 @@ Pulsar Node.js producers have the following methods available: | :-------- | :---------- | :------ | | `topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer publishes messages. | | | `producerName` | A name for the producer. If you do not explicitly assign a name, Pulsar automatically generates a globally unique name. If you choose to explicitly assign a name, it needs to be unique across *all* Pulsar clusters, otherwise the creation operation throws an error. | | -| `sendTimeoutMs` | When publishing a message to a topic, the producer waits for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error is thrown. If you set `sendTimeoutMs` to -1, the timeout is set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30000 | +| `sendTimeoutMs` | When publishing a message to a topic, the producer waits for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error is thrown. If you set `sendTimeoutMs` to -1, the timeout is set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30000 | | `initialSequenceId` | The initial sequence ID of the message. When producer send message, add sequence ID to message. The ID is increased each time to send. | | | `maxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `send` method fails *unless* `blockIfQueueFull` is set to `true`. | 1000 | | `maxPendingMessagesAcrossPartitions` | The maximum size of the sum of partition's pending queue. | 50000 | diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-python.md b/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-python.md index 72aaf211ae2b2..90cc840daa0a8 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [Python directory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [Python directory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-websocket.md index 16eb49af81218..77ec54f803dc5 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/client-libraries-websocket.md @@ -5,14 +5,14 @@ sidebar_label: "WebSocket" original_id: client-libraries-websocket --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-architecture-overview.md index 95cf02eac109c..4baa8c30a0d00 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-architecture-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Architecture" original_id: concepts-architecture-overview --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -17,20 +17,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers. The producers connect to the brokers to publish messages and the consumers connect to the brokers to consume the messages. -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -40,9 +40,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -136,17 +136,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -154,7 +154,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-authentication.md b/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-authentication.md index b375ecb16a2d8..f6307890c904a 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-authentication.md @@ -5,5 +5,5 @@ sidebar_label: "Authentication and Authorization" original_id: concepts-authentication --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-clients.md b/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-clients.md index b68f76a2d8b08..4040624f7d636 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-clients.md @@ -5,12 +5,12 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -38,7 +38,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-messaging.md b/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-messaging.md index a0f7d046badc7..dbe540c5df8e2 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-messaging.md @@ -23,8 +23,8 @@ Messages are the basic "unit" of Pulsar. The following table lists the component Component | Description :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer who produces the message. If you do not specify a producer name, the default name is used. Sequence ID | Each Pulsar message belongs to an ordered sequence on its topic. The sequence ID of the message is its order in that sequence. @@ -52,7 +52,7 @@ The default size of a message is 5 MB. You can configure the max size of a messa ``` -> For more information on Pulsar messages, see Pulsar [binary protocol](developing-binary-protocol). +> For more information on Pulsar messages, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -149,7 +149,7 @@ Messages are received from [brokers](reference-terminology.md#broker) either syn ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -502,7 +502,7 @@ Decisions about routing and subscription modes can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription modes work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -514,7 +514,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -552,7 +552,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -639,7 +639,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -662,12 +662,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication.md). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-multi-tenancy.md index 730458c904618..93a59557b2efc 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-multi-tenancy.md @@ -5,7 +5,7 @@ sidebar_label: "Multi Tenancy" original_id: concepts-multi-tenancy --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -21,7 +21,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -29,7 +29,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-overview.md b/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-overview.md index 310a78dff4565..c643aa0ce7bbc 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription modes](concepts-messaging.md#subscription-modes) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-replication.md b/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-replication.md index 6e23962e99771..799f0eb4d92c6 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-replication.md @@ -5,5 +5,5 @@ sidebar_label: "Geo Replication" original_id: concepts-replication --- -Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that. +Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo.md) in Pulsar enables you to do that. diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-tiered-storage.md index 0b45b0a2ed501..f6988e53a8cd4 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-topic-compaction.md index e402c965384fd..34b7ed7fbbd31 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/concepts-topic-compaction.md @@ -7,7 +7,7 @@ original_id: concepts-topic-compaction Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -24,7 +24,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.9.1-deprecated/cookbooks-compaction.md index 6fcbc197f9ecb..dfa314727241a 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.9.1-deprecated/cookbooks-non-persistent.md index 391569a8cbc6b..178301e86eb8d 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/cookbooks-non-persistent.md @@ -41,7 +41,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -55,7 +55,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.9.1-deprecated/cookbooks-partitioned.md index 7882fb9025312..fb9ac354cc6d6 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/cookbooks-partitioned.md @@ -4,4 +4,4 @@ title: Partitioned topics sidebar_label: "Partitioned Topics" original_id: cookbooks-partitioned --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.9.1-deprecated/cookbooks-retention-expiry.md index 0283a64312ca7..c8c46b3caa1be 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/cookbooks-retention-expiry.md @@ -22,7 +22,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-aws.md b/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-aws.md index 071effeffde24..93c389b56e2cf 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-aws.md @@ -5,7 +5,7 @@ sidebar_label: "Amazon Web Services" original_id: deploy-aws --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -210,7 +210,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors) , edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md) , edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-bare-metal-multi-cluster.md index df25604af4e62..f25b11041c5e3 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-bare-metal-multi-cluster.md @@ -7,13 +7,13 @@ original_id: deploy-bare-metal-multi-cluster :::tip -1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster). -2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster.md). +2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo).Deploying a multi-cluster Pulsar instance consists of the following steps: +A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md).Deploying a multi-cluster Pulsar instance consists of the following steps: 1. Deploying two separate ZooKeeper quorums: a local quorum for each cluster in the instance and a configuration store quorum for instance-wide tasks 2. Initializing cluster metadata for each cluster @@ -22,7 +22,7 @@ A Pulsar instance consists of multiple Pulsar clusters working in unison. You ca > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. ## System requirement @@ -65,7 +65,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -244,7 +244,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -363,16 +363,16 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. -You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > **Service discovery already provided by many scheduling systems** -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-bare-metal.md index ed0c9d7da465b..9bb4235cece5d 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-bare-metal.md @@ -7,9 +7,9 @@ original_id: deploy-bare-metal :::tip -1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster). -2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster.md). +2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -126,7 +126,7 @@ The extracted directory contains the following subdirectories: Directory | Contains :---------|:-------- -`bin` |[command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` |[command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `data` | The data storage directory that ZooKeeper and BookKeeper use `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -279,9 +279,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (you had better not use a different port). -`--web-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). +`--web-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should not use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (you had better not use a different port). -`--broker-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). +`--broker-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). > If you do not have a DNS server, you can use multi-host format in the service URL with the following settings: @@ -421,7 +421,7 @@ webServicePortTls=8443 ### Enable Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -439,7 +439,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker). +If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker.md). ### Start Brokers diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-dcos.md b/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-dcos.md index 49cf624ea24f1..35a0a83d716ad 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -To enable all built-in [Pulsar IO](io-overview) connectors in your Pulsar deployment, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). +To enable all built-in [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-docker.md b/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-docker.md index 64a0939aa787b..8348d78deb237 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-docker.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-docker.md @@ -57,4 +57,4 @@ docker network connect pulsar broker To check whether the containers are successfully connected to the network, enter the `docker network inspect pulsar` command. -For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal). +For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal.md). diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-kubernetes.md index dc7123d000467..1aefc6ad79f71 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-kubernetes.md @@ -6,6 +6,6 @@ original_id: deploy-kubernetes --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-monitoring.md b/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-monitoring.md index 87d435050df27..2b5c19344dc8c 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/deploy-monitoring.md @@ -113,7 +113,7 @@ http://$FUNCTIONS_WORKER_ADDRESS:$WORKER_PORT/metrics: You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy). +When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy.md). ## Dashboards @@ -121,7 +121,7 @@ When you collect time series statistics, the major problem is to make sure the n ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager). +The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/develop-schema.md b/site2/website/versioned_docs/version-2.9.1-deprecated/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/develop-schema.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/functions-deploy.md b/site2/website/versioned_docs/version-2.9.1-deprecated/functions-deploy.md index d8fe0fc14e5bb..2a0d68d6c623c 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/functions-deploy.md @@ -9,12 +9,12 @@ original_id: functions-deploy To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface @@ -179,7 +179,7 @@ $ bin/pulsar-admin functions create \ Package management enables version management and simplifies the upgrade and rollback processes for Functions, Sinks, and Sources. When you use the same function, sink and source in different namespaces, you can upload them to a common package management system. -To use [Package management service](admin-api-packages), ensure that the package management service has been enabled in your cluster by setting the following properties in `broker.conf`. +To use [Package management service](admin-api-packages.md), ensure that the package management service has been enabled in your cluster by setting the following properties in `broker.conf`. > Note: Package management service is not enabled by default. diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/functions-develop.md b/site2/website/versioned_docs/version-2.9.1-deprecated/functions-develop.md index 36ea4b4a21073..2e29aa1c47400 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/functions-develop.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/functions-develop.md @@ -1144,7 +1144,7 @@ You can monitor Pulsar Functions that have been deployed with the following meth - Check the metrics provided by Pulsar. - Pulsar Functions expose the metrics that can be collected and used for monitoring the health of **Java, Python, and Go** functions. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar Functions expose the metrics that can be collected and used for monitoring the health of **Java, Python, and Go** functions. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. For the complete list of the function metrics, see [here](reference-metrics.md#pulsar-functions). @@ -1214,7 +1214,7 @@ Currently, the feature is not available in Go. ## Security -If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker). For more details, refer to [Security settings](functions-worker.md#security-settings). +If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings). Pulsar Functions can support the following providers: diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/functions-overview.md b/site2/website/versioned_docs/version-2.9.1-deprecated/functions-overview.md index 25dc6029acf70..816d301e0fd0e 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/functions-overview.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/functions-overview.md @@ -164,7 +164,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/functions-package.md b/site2/website/versioned_docs/version-2.9.1-deprecated/functions-package.md index 2666d5944b2e0..db2c4e987dc7b 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/functions-package.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/functions-package.md @@ -15,7 +15,7 @@ Currently, the window function is not available in Python and Go. ## Prerequisite -Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm). +Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm.md). To check whether the Docker image starts, you can use the `docker ps` command. diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/functions-worker.md b/site2/website/versioned_docs/version-2.9.1-deprecated/functions-worker.md index 78f4f19487951..49fc76b30bdaa 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/functions-worker.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/functions-worker.md @@ -4,7 +4,7 @@ title: Deploy and manage functions worker sidebar_label: "Setup: Pulsar Functions Worker" original_id: functions-worker --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -179,7 +179,7 @@ brokerClientTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -199,7 +199,7 @@ authenticationProviders: [ provider1, provider2 ] ``` For *TLS Authentication* provider, follow the example below to add the necessary settings. -See [TLS Authentication](security-tls-authentication) for more details. +See [TLS Authentication](security-tls-authentication.md) for more details. ``` @@ -223,7 +223,7 @@ properties: ``` For *Token Authentication* provider, add necessary settings for `properties` if needed. -See [Token Authentication](security-jwt) for more details. +See [Token Authentication](security-jwt.md) for more details. Note: key files must be DER-encoded ``` @@ -262,7 +262,7 @@ superUserRoles: You can use the public and private key pair that the application configures to perform encryption. Only the consumers with a valid key can decrypt the encrypted messages. -To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption). +To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption.md). We include the relevant configuration information of `CryptoConfig` into `ProducerConfig`. The specific configurable field information about `CryptoConfig` is as follows: diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/getting-started-docker.md b/site2/website/versioned_docs/version-2.9.1-deprecated/getting-started-docker.md index 71f1bb9734fb7..de5ead69e164b 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/getting-started-docker.md @@ -43,8 +43,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -103,7 +103,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/getting-started-helm.md b/site2/website/versioned_docs/version-2.9.1-deprecated/getting-started-helm.md index 98a3097d436db..5e9f7044a6d74 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/getting-started-helm.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/getting-started-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -409,7 +409,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/getting-started-pulsar.md b/site2/website/versioned_docs/version-2.9.1-deprecated/getting-started-pulsar.md index 11c5e66cb62c2..752590f57b558 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/getting-started-pulsar.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/getting-started-pulsar.md @@ -5,13 +5,13 @@ sidebar_label: "Pulsar 2.0" original_id: pulsar-2.0 --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/getting-started-standalone.md b/site2/website/versioned_docs/version-2.9.1-deprecated/getting-started-standalone.md index 436355aa832bc..9137ba291421a 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > **Pulsar in production?** -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -65,7 +65,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -74,7 +74,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -122,7 +122,7 @@ pulsar-io-aerospike-@pulsar:version@.nar :::note * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -135,7 +135,7 @@ pulsar-io-aerospike-@pulsar:version@.nar ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -168,12 +168,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -205,7 +205,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/helm-install.md b/site2/website/versioned_docs/version-2.9.1-deprecated/helm-install.md index 80fb7f8d6c8d6..9f81f52e0dab1 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/helm-install.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/helm-install.md @@ -21,7 +21,7 @@ Before deploying Pulsar, you need to prepare your environment. ### Tools -Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools) on your computer. +Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools.md) on your computer. ## Cloud cluster preparation @@ -31,8 +31,8 @@ To create and connect to the Kubernetes cluster, follow the instructions: ## Pulsar deployment -Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy). +Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy.md). ## Pulsar upgrade -To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade). +To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade.md). diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/helm-overview.md b/site2/website/versioned_docs/version-2.9.1-deprecated/helm-overview.md index fc4cbfe426a93..125f595cbe68a 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/helm-overview.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/helm-overview.md @@ -52,9 +52,9 @@ Moreover, Helm chart supports: ## Quick start -To run with Apache Pulsar Helm chart as fast as possible in a **non-production** use case, we provide a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +To run with Apache Pulsar Helm chart as fast as possible in a **non-production** use case, we provide a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -This guide walks you through deploying Apache Pulsar Helm chart with default values and features, but it is *not* suitable for deployments in production-ready environments. To deploy the charts in production under sustained load, you can follow the complete [Installation Guide](helm-install). +This guide walks you through deploying Apache Pulsar Helm chart with default values and features, but it is *not* suitable for deployments in production-ready environments. To deploy the charts in production under sustained load, you can follow the complete [Installation Guide](helm-install.md). ## Troubleshooting @@ -64,7 +64,7 @@ Although we have done our best to make these charts as seamless as possible, tro The Apache Pulsar Helm chart contains all required dependencies. -If you deploy a PoC for testing, we strongly suggest you follow this [Quick Start Guide](getting-started-helm) for your first iteration. +If you deploy a PoC for testing, we strongly suggest you follow this [Quick Start Guide](getting-started-helm.md) for your first iteration. 1. [Preparation](helm-prepare.md) 2. [Deployment](helm-deploy.md) @@ -82,7 +82,7 @@ helm upgrade apache/pulsar -f pulsar.yaml ``` -For more detailed information, see [Upgrading](helm-upgrade). +For more detailed information, see [Upgrading](helm-upgrade.md). ## Uninstallation diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/helm-tools.md b/site2/website/versioned_docs/version-2.9.1-deprecated/helm-tools.md index efd61eae32506..6ba89006913b6 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/helm-tools.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/helm-tools.md @@ -25,7 +25,7 @@ You can get Helm from the project's [releases page](https://github.com/helm/helm ### Next steps -Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare). +Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare.md). ## Additional information diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/helm-upgrade.md b/site2/website/versioned_docs/version-2.9.1-deprecated/helm-upgrade.md index e39cf97407355..7d671e6bfb3c1 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/helm-upgrade.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/helm-upgrade.md @@ -20,7 +20,7 @@ You can retrieve your previous `--set` arguments cleanly, with `helm get values To upgrade Apache Pulsar to a newer version, follow these steps: 1. Check the change log for the specific version you would like to upgrade to. -2. Go through [deployment documentation](helm-deploy) step by step. +2. Go through [deployment documentation](helm-deploy.md) step by step. 3. Extract your previous `--set` arguments with the following command. ```bash diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/io-cdc.md b/site2/website/versioned_docs/version-2.9.1-deprecated/io-cdc.md index 20f16ae0d6904..e6e662884826d 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/io-cdc.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/io-cdc.md @@ -13,8 +13,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/io-develop.md b/site2/website/versioned_docs/version-2.9.1-deprecated/io-develop.md index a20206fbf6a06..d6f4f8261ac82 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/io-develop.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/io-develop.md @@ -14,15 +14,15 @@ import TabItem from '@theme/TabItem'; This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -275,7 +275,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -304,7 +304,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). @@ -379,7 +379,7 @@ Pulsar connectors enable you to move data in and out of Pulsar easily. It is imp - Check the metrics provided by Pulsar. - Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. - Set and check your customized metrics. diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/io-overview.md b/site2/website/versioned_docs/version-2.9.1-deprecated/io-overview.md index 0070a015296db..3db5ee34042d3 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/io-overview.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/io-overview.md @@ -158,7 +158,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/io-quickstart.md b/site2/website/versioned_docs/version-2.9.1-deprecated/io-quickstart.md index 00bcba3769ff7..40eaf5c1de221 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you are able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you are able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However, all the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -109,7 +109,7 @@ This section demonstrates how to connect Pulsar to Cassandra. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink.md). ::: @@ -236,11 +236,11 @@ You can create a configuration file through one of the following methods. ``` -For more information, see [Cassandra sink connector](io-cassandra-sink). +For more information, see [Cassandra sink connector](io-cassandra-sink.md). ### Create a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on them. Run the following command to create a Cassandra sink connector with sink type _cassandra_ and the config file _examples/cassandra-sink.yml_ created previously. @@ -267,7 +267,7 @@ as a Pulsar Function and writes the messages produced in the topic _test_cassand ### Inspect a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * Get the information of a Cassandra sink. @@ -431,7 +431,7 @@ to monitor a connector and perform other operations on it. ### Delete a Cassandra Sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. ```bash @@ -454,7 +454,7 @@ This section demonstrates how to connect Pulsar to PostgreSQL. ::: ->For more information, see [JDBC sink connector](io-jdbc-sink). +>For more information, see [JDBC sink connector](io-jdbc-sink.md). ### Setup a PostgreSQL cluster @@ -614,7 +614,7 @@ In this section, you need to configure a JDBC sink connector. ### Create a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on it. This example creates a sink connector and specifies the desired information. @@ -660,7 +660,7 @@ The sink has been created successfully if the following message appears. ### Inspect a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * List all running JDBC sink(s). @@ -780,7 +780,7 @@ to monitor a connector and perform other operations on it. ### Stop a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to stop a connector and perform other operations on it. ```bash @@ -808,7 +808,7 @@ The sink instance has been stopped successfully if the following message disappe ### Restart a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to restart a connector and perform other operations on it. ```bash @@ -844,7 +844,7 @@ Note that `pulsar-admin sinks localrun options` **runs a sink connector locally* ### Update a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to update a connector and perform other operations on it. This example updates the parallelism of the _pulsar-postgres-jdbc-sink_ sink connector to 2. @@ -912,7 +912,7 @@ The result shows that the parallelism is 2. ### Delete a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. This example deletes the _pulsar-postgres-jdbc-sink_ sink connector. diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/io-use.md b/site2/website/versioned_docs/version-2.9.1-deprecated/io-use.md index b6409d4420ad7..da9ed746c4d37 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/io-use.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/io-use.md @@ -15,7 +15,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/kubernetes-helm.md b/site2/website/versioned_docs/version-2.9.1-deprecated/kubernetes-helm.md index a9f768e6a96b3..ea92a0968cd7d 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/kubernetes-helm.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/kubernetes-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -403,7 +403,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/reference-cli-tools.md b/site2/website/versioned_docs/version-2.9.1-deprecated/reference-cli-tools.md index eff2625277504..6893da3ec6394 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/reference-configuration.md b/site2/website/versioned_docs/version-2.9.1-deprecated/reference-configuration.md index 8ef9685095c0d..d71ce8214b21d 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/reference-configuration.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/reference-configuration.md @@ -8,7 +8,7 @@ original_id: reference-configuration -You can manage Pulsar configuration by configuration files in the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a Pulsar [installation](getting-started-standalone). +You can manage Pulsar configuration by configuration files in the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a Pulsar [installation](getting-started-standalone.md). - [BookKeeper](#bookkeeper) - [Broker](#broker) diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/schema-get-started.md b/site2/website/versioned_docs/version-2.9.1-deprecated/schema-get-started.md index 97e270586a26d..afacb0fa51f2e 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/schema-get-started.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/schema-get-started.md @@ -17,7 +17,7 @@ Applications typically adopt one of the following approaches to guarantee type s #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ### Client-side approach diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/schema-manage.md b/site2/website/versioned_docs/version-2.9.1-deprecated/schema-manage.md index 4ae363882b2d5..c588aae619eee 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/schema-manage.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/schema-manage.md @@ -160,7 +160,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/schema-understand.md b/site2/website/versioned_docs/version-2.9.1-deprecated/schema-understand.md index f454dd6873e8e..55bc662c66633 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/schema-understand.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/schema-understand.md @@ -479,7 +479,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```java diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/security-athenz.md b/site2/website/versioned_docs/version-2.9.1-deprecated/security-athenz.md index 947c3f470be46..8a39fe25316d0 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/security-athenz.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/security-athenz.md @@ -76,7 +76,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/security-authorization.md b/site2/website/versioned_docs/version-2.9.1-deprecated/security-authorization.md index cc9c03d5fc1dd..9cfd7c8c203f6 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/security-authorization.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/security-authorization.md @@ -27,7 +27,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -59,7 +59,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -87,7 +87,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication @@ -117,7 +117,7 @@ PulsarAdmin admin = PulsarAdmin.builder() When a client is identified with multiple roles in a token (the type of role claim in the token is an array) during the authentication process, Pulsar supports to check the permissions of all the roles and further authorize the client as long as one of its roles has the required permissions. > **Note**
    -> This authorization method is only compatible with [JWT authentication](security-jwt). +> This authorization method is only compatible with [JWT authentication](security-jwt.md). To enable this authorization method, configure the authorization provider as `MultiRolesTokenAuthorizationProvider` in the `conf/broker.conf` file. diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/security-bouncy-castle.md b/site2/website/versioned_docs/version-2.9.1-deprecated/security-bouncy-castle.md index abc25c71b8ae2..be937055d8e31 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/security-bouncy-castle.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/security-bouncy-castle.md @@ -16,7 +16,7 @@ In Pulsar, security and crypto have dependencies on BouncyCastle Jars. For the d `Bouncy Castle` provides both [FIPS](https://www.bouncycastle.org/fips_faq.html) and non-FIPS version. But in a JVM, you can not include both of the 2 versions, and you need to exclude the current version before include the other. -In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. +In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption.md). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. ## How BouncyCastle modules packaged in Pulsar diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/security-extending.md b/site2/website/versioned_docs/version-2.9.1-deprecated/security-extending.md index c088e3a2c1c2c..e7484453b8beb 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/security-extending.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/security-extending.md @@ -9,7 +9,7 @@ Pulsar provides a way to use custom authentication and authorization mechanisms. ## Authentication -Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview). +Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview.md). You can use a custom authentication mechanism by providing the implementation in the form of two plugins. One plugin is for the Client library and the other plugin is for the Pulsar Proxy and/or Pulsar Broker to validate the credentials. diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/security-jwt.md b/site2/website/versioned_docs/version-2.9.1-deprecated/security-jwt.md index 05e641ed67c17..1fa65b7c27f60 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/security-jwt.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/security-jwt.md @@ -32,11 +32,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -305,7 +305,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/security-kerberos.md b/site2/website/versioned_docs/version-2.9.1-deprecated/security-kerberos.md index 670586a9d1691..c49fa3bea1fce 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/security-kerberos.md @@ -373,7 +373,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/security-overview.md b/site2/website/versioned_docs/version-2.9.1-deprecated/security-overview.md index f8e8eac98f805..227dd5b5d4fc3 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/security-overview.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/security-overview.md @@ -21,7 +21,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/security-tls-authentication.md b/site2/website/versioned_docs/version-2.9.1-deprecated/security-tls-authentication.md index f3c32a05d9751..85d2240f41306 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/security-tls-authentication.md @@ -7,9 +7,9 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -103,7 +103,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -123,7 +123,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/security-tls-keystore.md b/site2/website/versioned_docs/version-2.9.1-deprecated/security-tls-keystore.md index 6cc3a2c60835e..0b3b50fcebb10 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/security-tls-keystore.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/security-tls-keystore.md @@ -7,7 +7,7 @@ original_id: security-tls-keystore ## Overview -Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication) between clients and Apache Pulsar service. +Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication.md) between clients and Apache Pulsar service. By default it uses PEM format file configuration. This page tries to describe use [KeyStore](https://en.wikipedia.org/wiki/Java_KeyStore) type configure for TLS. @@ -185,7 +185,7 @@ This is similar to [TLS encryption configuing for client with PEM type](security For a a minimal configuration, user need to provide the TrustStore information. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties @@ -278,7 +278,7 @@ webSocketServiceEnabled=false Besides the TLS encryption configuring. The main work is configuring the KeyStore, which contains a valid CN as client role, for client. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/security-tls-transport.md b/site2/website/versioned_docs/version-2.9.1-deprecated/security-tls-transport.md index 92e8dc3c3d280..2cad17a78c350 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,9 +27,9 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). -`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ## Create TLS certificates @@ -130,7 +130,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Configure broker -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -202,7 +202,7 @@ The examples below show that hostname verification is disabled for the CLI tools ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/security-token-admin.md b/site2/website/versioned_docs/version-2.9.1-deprecated/security-token-admin.md index 1a438dc7923ee..a265f6320d28f 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/security-token-admin.md @@ -163,7 +163,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.9.1-deprecated/sql-deployment-configurations.md index 67f7ab17c77d0..02d8bc78f6cb9 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/sql-deployment-configurations.md @@ -64,7 +64,7 @@ Since Pulsar SQL is powered by [Trino (formerly Presto SQL)](https://trino.io), :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/sql-overview.md b/site2/website/versioned_docs/version-2.9.1-deprecated/sql-overview.md index 4a4d5f01b0bc5..0235d0256c5f1 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/sql-overview.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/sql-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Overview" original_id: sql-overview --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL)](https://trino.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL.md)](https://trino.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/standalone-docker.md b/site2/website/versioned_docs/version-2.9.1-deprecated/standalone-docker.md index 6ff3696bd3a44..1710ec819d7a4 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/standalone-docker.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/standalone-docker.md @@ -46,8 +46,8 @@ For more information, see [Topics](concepts-messaging.md#topics). ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -106,7 +106,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/standalone.md b/site2/website/versioned_docs/version-2.9.1-deprecated/standalone.md index 32970bfcfe24f..9bff74d92c10b 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/standalone.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > **Pulsar in production?** -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -65,7 +65,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -74,7 +74,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -122,7 +122,7 @@ pulsar-io-aerospike-@pulsar:version@.nar :::note * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -135,7 +135,7 @@ pulsar-io-aerospike-@pulsar:version@.nar ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -168,12 +168,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -205,7 +205,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.9.1-deprecated/tiered-storage-overview.md b/site2/website/versioned_docs/version-2.9.1-deprecated/tiered-storage-overview.md index 6c22c2214930a..bebd16d28a098 100644 --- a/site2/website/versioned_docs/version-2.9.1-deprecated/tiered-storage-overview.md +++ b/site2/website/versioned_docs/version-2.9.1-deprecated/tiered-storage-overview.md @@ -13,9 +13,9 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws). + - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws.md). - - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs). + - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs.md). ::: @@ -25,7 +25,7 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem). + For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem.md). ::: diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/admin-api-brokers.md b/site2/website/versioned_docs/version-2.9.2-deprecated/admin-api-brokers.md index 22409e6c23893..930fe69ecfb0e 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/admin-api-brokers.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/admin-api-brokers.md @@ -176,7 +176,7 @@ One way to configure a Pulsar [broker](reference-terminology.md#broker) is to su But since all broker configuration in Pulsar is stored in ZooKeeper, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values. -* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. +* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. * In the Pulsar admin {@inject: rest:REST:/} API, dynamic configuration is managed through the `/admin/v2/brokers/configuration` endpoint. ### Update dynamic configuration diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/admin-api-clusters.md b/site2/website/versioned_docs/version-2.9.2-deprecated/admin-api-clusters.md index b2f0fa3b25670..1d0c5dc9786f5 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/admin-api-clusters.md @@ -89,7 +89,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -113,11 +113,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block > - For the latest and complete information about `Java admin API`, including classes, methods, descriptions, and more, see [Java admin API doc](https://pulsar.apache.org/api/admin/). -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -118,7 +118,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-cgo.md b/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-cgo.md index c79f7bb965473..f352f942b7714 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-cgo.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-cgo.md @@ -24,7 +24,7 @@ Currently, the following Go clients are maintained in two repositories. ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Install go package @@ -57,7 +57,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -75,7 +75,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -236,14 +236,14 @@ Parameter | Description | Default `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | `Properties`| Attach a set of application defined properties to the producer. This properties will be visible in the topic stats | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | Set the number of max pending messages across all the partitions. This setting will be used to lower the max pending messages for each partition `MaxPendingMessages(int)`, if the total exceeds the configured value.| `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | `Batching` | Control whether automatic batching of messages is enabled for the producer. | false `BatchingMaxPublishDelay` | Set the time period within which the messages sent will be batched (default: 1ms) if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or until | 1ms `BatchingMaxMessages` | Set the maximum number of messages permitted in a batch. (default: 1000) If set to a value greater than 1, messages will be queued until this threshold is reached or batch interval has elapsed | 1000 @@ -512,7 +512,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-cpp.md index 3c94293b7929c..455cf02116d50 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-cpp.md @@ -602,7 +602,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema This section describes some examples about schema. For more information about -schema, see [Pulsar schema](schema-get-started). +schema, see [Pulsar schema](schema-get-started.md). ### Avro schema diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-dotnet.md b/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-dotnet.md index ade664cd95277..b574fa0b2e5ed 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-dotnet.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-dotnet.md @@ -170,7 +170,7 @@ var client = PulsarClient.Builder() Currently, the Pulsar C# client supports the TLS (Transport Layer Security) and JWT (JSON Web Token) authentication. -If you have followed [Authentication using TLS](security-tls-authentication), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: +If you have followed [Authentication using TLS](security-tls-authentication.md), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: 1. Create an unencrypted and password-less pfx file. diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-go.md b/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-go.md index fbb2573aa5db8..6281b03dd8c80 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-go.md @@ -35,7 +35,7 @@ import "github.com/apache/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -61,7 +61,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -844,7 +844,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker @@ -864,7 +864,7 @@ opts := pulsar.ClientOptions{ ## OAuth2 authentication -To use [OAuth2 authentication](security-oauth2), you'll need to configure your client to perform the following operations. +To use [OAuth2 authentication](security-oauth2.md), you'll need to configure your client to perform the following operations. This example shows how to configure OAuth2 authentication. ```go diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-java.md b/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-java.md index e87c5e6c23c8a..c3d41a3f13da2 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-java.md @@ -5,7 +5,7 @@ sidebar_label: "Java" original_id: client-libraries-java --- -You can use a Pulsar Java client to create the Java [producer](#producer), [consumer](#consumer), and [readers](#reader) of messages and to perform [administrative tasks](admin-api-overview). The current Java client version is **@pulsar:version@**. +You can use a Pulsar Java client to create the Java [producer](#producer), [consumer](#consumer), and [readers](#reader) of messages and to perform [administrative tasks](admin-api-overview.md). The current Java client version is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), and [reader](#reader) of a Java client are thread-safe. @@ -14,10 +14,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Include both `pulsar-client` and `pulsar-client-admin`
    Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.
    In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -57,7 +57,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -83,7 +83,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -114,7 +114,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -240,7 +240,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics cookbook](cookbooks-partitioned). +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics cookbook](cookbooks-partitioned.md). ### Async send @@ -827,7 +827,7 @@ Total hash range size is 65536, so the max end of the range should be less than ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producer) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producer) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -945,7 +945,7 @@ For example of ProtobufNativeSchema, see [`SchemaDefinition` in `Complex type`]( ## Authentication -Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2). You can use the Pulsar Java client with all of them. +Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2.md). You can use the Pulsar Java client with all of them. ### TLS Authentication @@ -972,7 +972,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` @@ -1009,7 +1009,7 @@ PulsarClient client = PulsarClient.builder() ### Oauth2 -The following example shows how to use [Oauth2](security-oauth2) as an authentication provider for the Pulsar Java client. +The following example shows how to use [Oauth2](security-oauth2.md) as an authentication provider for the Pulsar Java client. You can use the factory method to configure authentication for Pulsar Java client. diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-node.md b/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-node.md index e6584ede93f6d..1ff37b2629466 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-node.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-node.md @@ -48,7 +48,7 @@ Also, this library works only in Node.js 10.x or later because it uses the [`nod ::: ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here is an example for `localhost`: @@ -66,7 +66,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication), the URL looks like this: +If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication.md), the URL looks like this: ```http @@ -101,7 +101,7 @@ The following configurable parameters are available for Pulsar clients: | Parameter | Description | Default | | :-------- | :---------- | :------ | | `serviceUrl` | The connection URL for the Pulsar cluster. See [above](#connection-urls) for more info. | | -| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication) for more info. | | +| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication.md) for more info. | | | `operationTimeoutSeconds` | The timeout for Node.js client operations (creating producers, subscribing to and unsubscribing from [topics](reference-terminology.md#topic)). Retries occur until this threshold is reached, at which point the operation fails. | 30 | | `ioThreads` | The number of threads to use for handling connections to Pulsar [brokers](reference-terminology.md#broker). | 1 | | `messageListenerThreads` | The number of threads used by message listeners ([consumers](#consumers) and [readers](#readers)). | 1 | @@ -154,7 +154,7 @@ Pulsar Node.js producers have the following methods available: | :-------- | :---------- | :------ | | `topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer publishes messages. | | | `producerName` | A name for the producer. If you do not explicitly assign a name, Pulsar automatically generates a globally unique name. If you choose to explicitly assign a name, it needs to be unique across *all* Pulsar clusters, otherwise the creation operation throws an error. | | -| `sendTimeoutMs` | When publishing a message to a topic, the producer waits for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error is thrown. If you set `sendTimeoutMs` to -1, the timeout is set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30000 | +| `sendTimeoutMs` | When publishing a message to a topic, the producer waits for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error is thrown. If you set `sendTimeoutMs` to -1, the timeout is set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30000 | | `initialSequenceId` | The initial sequence ID of the message. When producer send message, add sequence ID to message. The ID is increased each time to send. | | | `maxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `send` method fails *unless* `blockIfQueueFull` is set to `true`. | 1000 | | `maxPendingMessagesAcrossPartitions` | The maximum size of the sum of partition's pending queue. | 50000 | diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-python.md b/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-python.md index 72aaf211ae2b2..90cc840daa0a8 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [Python directory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [Python directory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-websocket.md index 16eb49af81218..77ec54f803dc5 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/client-libraries-websocket.md @@ -5,14 +5,14 @@ sidebar_label: "WebSocket" original_id: client-libraries-websocket --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-architecture-overview.md index 95cf02eac109c..4baa8c30a0d00 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-architecture-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Architecture" original_id: concepts-architecture-overview --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -17,20 +17,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers. The producers connect to the brokers to publish messages and the consumers connect to the brokers to consume the messages. -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -40,9 +40,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -136,17 +136,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -154,7 +154,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-authentication.md b/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-authentication.md index b375ecb16a2d8..f6307890c904a 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-authentication.md @@ -5,5 +5,5 @@ sidebar_label: "Authentication and Authorization" original_id: concepts-authentication --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-clients.md b/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-clients.md index b68f76a2d8b08..4040624f7d636 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-clients.md @@ -5,12 +5,12 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -38,7 +38,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-messaging.md b/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-messaging.md index da65d508392d2..c7469606f3055 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-messaging.md @@ -23,8 +23,8 @@ Messages are the basic "unit" of Pulsar. The following table lists the component Component | Description :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer who produces the message. If you do not specify a producer name, the default name is used. Sequence ID | Each Pulsar message belongs to an ordered sequence on its topic. The sequence ID of the message is its order in that sequence. @@ -52,7 +52,7 @@ The default size of a message is 5 MB. You can configure the max size of a messa ``` -> For more information on Pulsar messages, see Pulsar [binary protocol](developing-binary-protocol). +> For more information on Pulsar messages, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -149,7 +149,7 @@ Messages are received from [brokers](reference-terminology.md#broker) either syn ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -502,7 +502,7 @@ Decisions about routing and subscription modes can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription modes work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -514,7 +514,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -552,7 +552,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -638,7 +638,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -661,12 +661,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication.md). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-multi-tenancy.md index 730458c904618..93a59557b2efc 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-multi-tenancy.md @@ -5,7 +5,7 @@ sidebar_label: "Multi Tenancy" original_id: concepts-multi-tenancy --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -21,7 +21,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -29,7 +29,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-overview.md b/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-overview.md index 310a78dff4565..c643aa0ce7bbc 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription modes](concepts-messaging.md#subscription-modes) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-replication.md b/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-replication.md index 6e23962e99771..799f0eb4d92c6 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-replication.md @@ -5,5 +5,5 @@ sidebar_label: "Geo Replication" original_id: concepts-replication --- -Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that. +Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo.md) in Pulsar enables you to do that. diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-tiered-storage.md index 0b45b0a2ed501..f6988e53a8cd4 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-topic-compaction.md index e402c965384fd..34b7ed7fbbd31 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/concepts-topic-compaction.md @@ -7,7 +7,7 @@ original_id: concepts-topic-compaction Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -24,7 +24,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.9.2-deprecated/cookbooks-compaction.md index 6fcbc197f9ecb..dfa314727241a 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.9.2-deprecated/cookbooks-non-persistent.md index 391569a8cbc6b..178301e86eb8d 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/cookbooks-non-persistent.md @@ -41,7 +41,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -55,7 +55,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.9.2-deprecated/cookbooks-partitioned.md index 7882fb9025312..fb9ac354cc6d6 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/cookbooks-partitioned.md @@ -4,4 +4,4 @@ title: Partitioned topics sidebar_label: "Partitioned Topics" original_id: cookbooks-partitioned --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.9.2-deprecated/cookbooks-retention-expiry.md index 0283a64312ca7..c8c46b3caa1be 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/cookbooks-retention-expiry.md @@ -22,7 +22,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-aws.md b/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-aws.md index 7ae3bb0fbb8b7..0d94fc13cdb2d 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-aws.md @@ -5,7 +5,7 @@ sidebar_label: "Amazon Web Services" original_id: deploy-aws --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -210,7 +210,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors) , edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md) , edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-bare-metal-multi-cluster.md index 1af30a644abe8..49fd3938c64d4 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-bare-metal-multi-cluster.md @@ -7,13 +7,13 @@ original_id: deploy-bare-metal-multi-cluster :::tip -1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster). -2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster.md). +2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo).Deploying a multi-cluster Pulsar instance consists of the following steps: +A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md).Deploying a multi-cluster Pulsar instance consists of the following steps: 1. Deploying two separate ZooKeeper quorums: a local quorum for each cluster in the instance and a configuration store quorum for instance-wide tasks 2. Initializing cluster metadata for each cluster @@ -22,7 +22,7 @@ A Pulsar instance consists of multiple Pulsar clusters working in unison. You ca > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. ## System requirement @@ -65,7 +65,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -244,7 +244,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -363,16 +363,16 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. -You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > **Service discovery already provided by many scheduling systems** -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-bare-metal.md index 6761a7c8fb110..292157e3ddc89 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-bare-metal.md @@ -7,9 +7,9 @@ original_id: deploy-bare-metal :::tip -1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster). -2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster.md). +2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -126,7 +126,7 @@ The extracted directory contains the following subdirectories: Directory | Contains :---------|:-------- -`bin` |[command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` |[command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `data` | The data storage directory that ZooKeeper and BookKeeper use `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -279,9 +279,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (you had better not use a different port). -`--web-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). +`--web-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should not use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (you had better not use a different port). -`--broker-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). +`--broker-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). > If you do not have a DNS server, you can use multi-host format in the service URL with the following settings: @@ -421,7 +421,7 @@ webServicePortTls=8443 ### Enable Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -439,7 +439,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker). +If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker.md). ### Start Brokers diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-dcos.md b/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-dcos.md index 49cf624ea24f1..35a0a83d716ad 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -To enable all built-in [Pulsar IO](io-overview) connectors in your Pulsar deployment, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). +To enable all built-in [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-docker.md b/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-docker.md index 64a0939aa787b..8348d78deb237 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-docker.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-docker.md @@ -57,4 +57,4 @@ docker network connect pulsar broker To check whether the containers are successfully connected to the network, enter the `docker network inspect pulsar` command. -For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal). +For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal.md). diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-kubernetes.md index dc7123d000467..1aefc6ad79f71 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-kubernetes.md @@ -6,6 +6,6 @@ original_id: deploy-kubernetes --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-monitoring.md b/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-monitoring.md index 87d435050df27..2b5c19344dc8c 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/deploy-monitoring.md @@ -113,7 +113,7 @@ http://$FUNCTIONS_WORKER_ADDRESS:$WORKER_PORT/metrics: You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy). +When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy.md). ## Dashboards @@ -121,7 +121,7 @@ When you collect time series statistics, the major problem is to make sure the n ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager). +The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/develop-schema.md b/site2/website/versioned_docs/version-2.9.2-deprecated/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/develop-schema.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/functions-deploy.md b/site2/website/versioned_docs/version-2.9.2-deprecated/functions-deploy.md index d8fe0fc14e5bb..2a0d68d6c623c 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/functions-deploy.md @@ -9,12 +9,12 @@ original_id: functions-deploy To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface @@ -179,7 +179,7 @@ $ bin/pulsar-admin functions create \ Package management enables version management and simplifies the upgrade and rollback processes for Functions, Sinks, and Sources. When you use the same function, sink and source in different namespaces, you can upload them to a common package management system. -To use [Package management service](admin-api-packages), ensure that the package management service has been enabled in your cluster by setting the following properties in `broker.conf`. +To use [Package management service](admin-api-packages.md), ensure that the package management service has been enabled in your cluster by setting the following properties in `broker.conf`. > Note: Package management service is not enabled by default. diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/functions-develop.md b/site2/website/versioned_docs/version-2.9.2-deprecated/functions-develop.md index 36ea4b4a21073..2e29aa1c47400 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/functions-develop.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/functions-develop.md @@ -1144,7 +1144,7 @@ You can monitor Pulsar Functions that have been deployed with the following meth - Check the metrics provided by Pulsar. - Pulsar Functions expose the metrics that can be collected and used for monitoring the health of **Java, Python, and Go** functions. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar Functions expose the metrics that can be collected and used for monitoring the health of **Java, Python, and Go** functions. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. For the complete list of the function metrics, see [here](reference-metrics.md#pulsar-functions). @@ -1214,7 +1214,7 @@ Currently, the feature is not available in Go. ## Security -If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker). For more details, refer to [Security settings](functions-worker.md#security-settings). +If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings). Pulsar Functions can support the following providers: diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/functions-overview.md b/site2/website/versioned_docs/version-2.9.2-deprecated/functions-overview.md index 25dc6029acf70..816d301e0fd0e 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/functions-overview.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/functions-overview.md @@ -164,7 +164,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/functions-package.md b/site2/website/versioned_docs/version-2.9.2-deprecated/functions-package.md index 2666d5944b2e0..db2c4e987dc7b 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/functions-package.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/functions-package.md @@ -15,7 +15,7 @@ Currently, the window function is not available in Python and Go. ## Prerequisite -Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm). +Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm.md). To check whether the Docker image starts, you can use the `docker ps` command. diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/functions-worker.md b/site2/website/versioned_docs/version-2.9.2-deprecated/functions-worker.md index d90b3456e6953..35e26926bb7ab 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/functions-worker.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/functions-worker.md @@ -4,7 +4,7 @@ title: Deploy and manage functions worker sidebar_label: "Setup: Pulsar Functions Worker" original_id: functions-worker --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -178,7 +178,7 @@ brokerClientTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -198,7 +198,7 @@ authenticationProviders: [ provider1, provider2 ] ``` For *TLS Authentication* provider, follow the example below to add the necessary settings. -See [TLS Authentication](security-tls-authentication) for more details. +See [TLS Authentication](security-tls-authentication.md) for more details. ``` @@ -222,7 +222,7 @@ properties: ``` For *Token Authentication* provider, add necessary settings for `properties` if needed. -See [Token Authentication](security-jwt) for more details. +See [Token Authentication](security-jwt.md) for more details. Note: key files must be DER-encoded ``` @@ -261,7 +261,7 @@ superUserRoles: You can use the public and private key pair that the application configures to perform encryption. Only the consumers with a valid key can decrypt the encrypted messages. -To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption). +To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption.md). We include the relevant configuration information of `CryptoConfig` into `ProducerConfig`. The specific configurable field information about `CryptoConfig` is as follows: diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/getting-started-docker.md b/site2/website/versioned_docs/version-2.9.2-deprecated/getting-started-docker.md index 71f1bb9734fb7..de5ead69e164b 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/getting-started-docker.md @@ -43,8 +43,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -103,7 +103,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/getting-started-helm.md b/site2/website/versioned_docs/version-2.9.2-deprecated/getting-started-helm.md index 98a3097d436db..5e9f7044a6d74 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/getting-started-helm.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/getting-started-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -409,7 +409,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/getting-started-pulsar.md b/site2/website/versioned_docs/version-2.9.2-deprecated/getting-started-pulsar.md index 11c5e66cb62c2..752590f57b558 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/getting-started-pulsar.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/getting-started-pulsar.md @@ -5,13 +5,13 @@ sidebar_label: "Pulsar 2.0" original_id: pulsar-2.0 --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/getting-started-standalone.md b/site2/website/versioned_docs/version-2.9.2-deprecated/getting-started-standalone.md index 90e3f820e74d1..573a33771d64e 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > **Pulsar in production?** -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -65,7 +65,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -74,7 +74,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -122,7 +122,7 @@ pulsar-io-aerospike-@pulsar:version@.nar :::note * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -135,7 +135,7 @@ pulsar-io-aerospike-@pulsar:version@.nar ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -168,12 +168,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -205,7 +205,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/helm-install.md b/site2/website/versioned_docs/version-2.9.2-deprecated/helm-install.md index 80fb7f8d6c8d6..9f81f52e0dab1 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/helm-install.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/helm-install.md @@ -21,7 +21,7 @@ Before deploying Pulsar, you need to prepare your environment. ### Tools -Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools) on your computer. +Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools.md) on your computer. ## Cloud cluster preparation @@ -31,8 +31,8 @@ To create and connect to the Kubernetes cluster, follow the instructions: ## Pulsar deployment -Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy). +Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy.md). ## Pulsar upgrade -To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade). +To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade.md). diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/helm-overview.md b/site2/website/versioned_docs/version-2.9.2-deprecated/helm-overview.md index fc4cbfe426a93..125f595cbe68a 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/helm-overview.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/helm-overview.md @@ -52,9 +52,9 @@ Moreover, Helm chart supports: ## Quick start -To run with Apache Pulsar Helm chart as fast as possible in a **non-production** use case, we provide a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +To run with Apache Pulsar Helm chart as fast as possible in a **non-production** use case, we provide a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -This guide walks you through deploying Apache Pulsar Helm chart with default values and features, but it is *not* suitable for deployments in production-ready environments. To deploy the charts in production under sustained load, you can follow the complete [Installation Guide](helm-install). +This guide walks you through deploying Apache Pulsar Helm chart with default values and features, but it is *not* suitable for deployments in production-ready environments. To deploy the charts in production under sustained load, you can follow the complete [Installation Guide](helm-install.md). ## Troubleshooting @@ -64,7 +64,7 @@ Although we have done our best to make these charts as seamless as possible, tro The Apache Pulsar Helm chart contains all required dependencies. -If you deploy a PoC for testing, we strongly suggest you follow this [Quick Start Guide](getting-started-helm) for your first iteration. +If you deploy a PoC for testing, we strongly suggest you follow this [Quick Start Guide](getting-started-helm.md) for your first iteration. 1. [Preparation](helm-prepare.md) 2. [Deployment](helm-deploy.md) @@ -82,7 +82,7 @@ helm upgrade apache/pulsar -f pulsar.yaml ``` -For more detailed information, see [Upgrading](helm-upgrade). +For more detailed information, see [Upgrading](helm-upgrade.md). ## Uninstallation diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/helm-tools.md b/site2/website/versioned_docs/version-2.9.2-deprecated/helm-tools.md index efd61eae32506..6ba89006913b6 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/helm-tools.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/helm-tools.md @@ -25,7 +25,7 @@ You can get Helm from the project's [releases page](https://github.com/helm/helm ### Next steps -Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare). +Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare.md). ## Additional information diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/helm-upgrade.md b/site2/website/versioned_docs/version-2.9.2-deprecated/helm-upgrade.md index e39cf97407355..7d671e6bfb3c1 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/helm-upgrade.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/helm-upgrade.md @@ -20,7 +20,7 @@ You can retrieve your previous `--set` arguments cleanly, with `helm get values To upgrade Apache Pulsar to a newer version, follow these steps: 1. Check the change log for the specific version you would like to upgrade to. -2. Go through [deployment documentation](helm-deploy) step by step. +2. Go through [deployment documentation](helm-deploy.md) step by step. 3. Extract your previous `--set` arguments with the following command. ```bash diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/io-cdc.md b/site2/website/versioned_docs/version-2.9.2-deprecated/io-cdc.md index 20f16ae0d6904..e6e662884826d 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/io-cdc.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/io-cdc.md @@ -13,8 +13,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/io-develop.md b/site2/website/versioned_docs/version-2.9.2-deprecated/io-develop.md index a20206fbf6a06..d6f4f8261ac82 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/io-develop.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/io-develop.md @@ -14,15 +14,15 @@ import TabItem from '@theme/TabItem'; This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -275,7 +275,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -304,7 +304,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). @@ -379,7 +379,7 @@ Pulsar connectors enable you to move data in and out of Pulsar easily. It is imp - Check the metrics provided by Pulsar. - Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. - Set and check your customized metrics. diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/io-overview.md b/site2/website/versioned_docs/version-2.9.2-deprecated/io-overview.md index 0070a015296db..3db5ee34042d3 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/io-overview.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/io-overview.md @@ -158,7 +158,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/io-quickstart.md b/site2/website/versioned_docs/version-2.9.2-deprecated/io-quickstart.md index 00bcba3769ff7..40eaf5c1de221 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you are able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you are able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However, all the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -109,7 +109,7 @@ This section demonstrates how to connect Pulsar to Cassandra. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink.md). ::: @@ -236,11 +236,11 @@ You can create a configuration file through one of the following methods. ``` -For more information, see [Cassandra sink connector](io-cassandra-sink). +For more information, see [Cassandra sink connector](io-cassandra-sink.md). ### Create a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on them. Run the following command to create a Cassandra sink connector with sink type _cassandra_ and the config file _examples/cassandra-sink.yml_ created previously. @@ -267,7 +267,7 @@ as a Pulsar Function and writes the messages produced in the topic _test_cassand ### Inspect a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * Get the information of a Cassandra sink. @@ -431,7 +431,7 @@ to monitor a connector and perform other operations on it. ### Delete a Cassandra Sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. ```bash @@ -454,7 +454,7 @@ This section demonstrates how to connect Pulsar to PostgreSQL. ::: ->For more information, see [JDBC sink connector](io-jdbc-sink). +>For more information, see [JDBC sink connector](io-jdbc-sink.md). ### Setup a PostgreSQL cluster @@ -614,7 +614,7 @@ In this section, you need to configure a JDBC sink connector. ### Create a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on it. This example creates a sink connector and specifies the desired information. @@ -660,7 +660,7 @@ The sink has been created successfully if the following message appears. ### Inspect a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * List all running JDBC sink(s). @@ -780,7 +780,7 @@ to monitor a connector and perform other operations on it. ### Stop a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to stop a connector and perform other operations on it. ```bash @@ -808,7 +808,7 @@ The sink instance has been stopped successfully if the following message disappe ### Restart a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to restart a connector and perform other operations on it. ```bash @@ -844,7 +844,7 @@ Note that `pulsar-admin sinks localrun options` **runs a sink connector locally* ### Update a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to update a connector and perform other operations on it. This example updates the parallelism of the _pulsar-postgres-jdbc-sink_ sink connector to 2. @@ -912,7 +912,7 @@ The result shows that the parallelism is 2. ### Delete a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. This example deletes the _pulsar-postgres-jdbc-sink_ sink connector. diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/io-use.md b/site2/website/versioned_docs/version-2.9.2-deprecated/io-use.md index b6409d4420ad7..da9ed746c4d37 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/io-use.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/io-use.md @@ -15,7 +15,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/reference-cli-tools.md b/site2/website/versioned_docs/version-2.9.2-deprecated/reference-cli-tools.md index eff2625277504..6893da3ec6394 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/reference-configuration.md b/site2/website/versioned_docs/version-2.9.2-deprecated/reference-configuration.md index 8ef9685095c0d..d71ce8214b21d 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/reference-configuration.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/reference-configuration.md @@ -8,7 +8,7 @@ original_id: reference-configuration -You can manage Pulsar configuration by configuration files in the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a Pulsar [installation](getting-started-standalone). +You can manage Pulsar configuration by configuration files in the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a Pulsar [installation](getting-started-standalone.md). - [BookKeeper](#bookkeeper) - [Broker](#broker) diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/schema-get-started.md b/site2/website/versioned_docs/version-2.9.2-deprecated/schema-get-started.md index 97e270586a26d..afacb0fa51f2e 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/schema-get-started.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/schema-get-started.md @@ -17,7 +17,7 @@ Applications typically adopt one of the following approaches to guarantee type s #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ### Client-side approach diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/schema-manage.md b/site2/website/versioned_docs/version-2.9.2-deprecated/schema-manage.md index 4ae363882b2d5..c588aae619eee 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/schema-manage.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/schema-manage.md @@ -160,7 +160,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/schema-understand.md b/site2/website/versioned_docs/version-2.9.2-deprecated/schema-understand.md index f454dd6873e8e..55bc662c66633 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/schema-understand.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/schema-understand.md @@ -479,7 +479,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```java diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/security-athenz.md b/site2/website/versioned_docs/version-2.9.2-deprecated/security-athenz.md index 947c3f470be46..8a39fe25316d0 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/security-athenz.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/security-athenz.md @@ -76,7 +76,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/security-authorization.md b/site2/website/versioned_docs/version-2.9.2-deprecated/security-authorization.md index cc9c03d5fc1dd..9cfd7c8c203f6 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/security-authorization.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/security-authorization.md @@ -27,7 +27,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -59,7 +59,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -87,7 +87,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication @@ -117,7 +117,7 @@ PulsarAdmin admin = PulsarAdmin.builder() When a client is identified with multiple roles in a token (the type of role claim in the token is an array) during the authentication process, Pulsar supports to check the permissions of all the roles and further authorize the client as long as one of its roles has the required permissions. > **Note**
    -> This authorization method is only compatible with [JWT authentication](security-jwt). +> This authorization method is only compatible with [JWT authentication](security-jwt.md). To enable this authorization method, configure the authorization provider as `MultiRolesTokenAuthorizationProvider` in the `conf/broker.conf` file. diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/security-bouncy-castle.md b/site2/website/versioned_docs/version-2.9.2-deprecated/security-bouncy-castle.md index abc25c71b8ae2..be937055d8e31 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/security-bouncy-castle.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/security-bouncy-castle.md @@ -16,7 +16,7 @@ In Pulsar, security and crypto have dependencies on BouncyCastle Jars. For the d `Bouncy Castle` provides both [FIPS](https://www.bouncycastle.org/fips_faq.html) and non-FIPS version. But in a JVM, you can not include both of the 2 versions, and you need to exclude the current version before include the other. -In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. +In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption.md). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. ## How BouncyCastle modules packaged in Pulsar diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/security-extending.md b/site2/website/versioned_docs/version-2.9.2-deprecated/security-extending.md index c088e3a2c1c2c..e7484453b8beb 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/security-extending.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/security-extending.md @@ -9,7 +9,7 @@ Pulsar provides a way to use custom authentication and authorization mechanisms. ## Authentication -Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview). +Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview.md). You can use a custom authentication mechanism by providing the implementation in the form of two plugins. One plugin is for the Client library and the other plugin is for the Pulsar Proxy and/or Pulsar Broker to validate the credentials. diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/security-jwt.md b/site2/website/versioned_docs/version-2.9.2-deprecated/security-jwt.md index 05e641ed67c17..1fa65b7c27f60 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/security-jwt.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/security-jwt.md @@ -32,11 +32,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -305,7 +305,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/security-kerberos.md b/site2/website/versioned_docs/version-2.9.2-deprecated/security-kerberos.md index 670586a9d1691..c49fa3bea1fce 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/security-kerberos.md @@ -373,7 +373,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/security-overview.md b/site2/website/versioned_docs/version-2.9.2-deprecated/security-overview.md index 23d4477214cb7..df630ab519bb4 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/security-overview.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/security-overview.md @@ -21,7 +21,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/security-tls-authentication.md b/site2/website/versioned_docs/version-2.9.2-deprecated/security-tls-authentication.md index f3c32a05d9751..85d2240f41306 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/security-tls-authentication.md @@ -7,9 +7,9 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -103,7 +103,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -123,7 +123,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/security-tls-keystore.md b/site2/website/versioned_docs/version-2.9.2-deprecated/security-tls-keystore.md index 6cc3a2c60835e..0b3b50fcebb10 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/security-tls-keystore.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/security-tls-keystore.md @@ -7,7 +7,7 @@ original_id: security-tls-keystore ## Overview -Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication) between clients and Apache Pulsar service. +Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication.md) between clients and Apache Pulsar service. By default it uses PEM format file configuration. This page tries to describe use [KeyStore](https://en.wikipedia.org/wiki/Java_KeyStore) type configure for TLS. @@ -185,7 +185,7 @@ This is similar to [TLS encryption configuing for client with PEM type](security For a a minimal configuration, user need to provide the TrustStore information. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties @@ -278,7 +278,7 @@ webSocketServiceEnabled=false Besides the TLS encryption configuring. The main work is configuring the KeyStore, which contains a valid CN as client role, for client. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/security-tls-transport.md b/site2/website/versioned_docs/version-2.9.2-deprecated/security-tls-transport.md index 92e8dc3c3d280..2cad17a78c350 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,9 +27,9 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). -`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ## Create TLS certificates @@ -130,7 +130,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Configure broker -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -202,7 +202,7 @@ The examples below show that hostname verification is disabled for the CLI tools ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/security-token-admin.md b/site2/website/versioned_docs/version-2.9.2-deprecated/security-token-admin.md index 1a438dc7923ee..a265f6320d28f 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/security-token-admin.md @@ -163,7 +163,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.9.2-deprecated/sql-deployment-configurations.md index 67f7ab17c77d0..02d8bc78f6cb9 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/sql-deployment-configurations.md @@ -64,7 +64,7 @@ Since Pulsar SQL is powered by [Trino (formerly Presto SQL)](https://trino.io), :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/sql-overview.md b/site2/website/versioned_docs/version-2.9.2-deprecated/sql-overview.md index 4a4d5f01b0bc5..0235d0256c5f1 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/sql-overview.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/sql-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Overview" original_id: sql-overview --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL)](https://trino.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL.md)](https://trino.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/standalone.md b/site2/website/versioned_docs/version-2.9.2-deprecated/standalone.md index 798ba379d2d18..a487be92adfcf 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/standalone.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > **Pulsar in production?** -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -65,7 +65,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -74,7 +74,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -122,7 +122,7 @@ pulsar-io-aerospike-@pulsar:version@.nar :::note * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -135,7 +135,7 @@ pulsar-io-aerospike-@pulsar:version@.nar ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -168,12 +168,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -205,7 +205,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.9.2-deprecated/tiered-storage-overview.md b/site2/website/versioned_docs/version-2.9.2-deprecated/tiered-storage-overview.md index 7c6528ce2b36c..c635034f463b4 100644 --- a/site2/website/versioned_docs/version-2.9.2-deprecated/tiered-storage-overview.md +++ b/site2/website/versioned_docs/version-2.9.2-deprecated/tiered-storage-overview.md @@ -13,9 +13,9 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws). + - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws.md). - - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs). + - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs.md). ::: @@ -25,7 +25,7 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem). + For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem.md). ::: diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/admin-api-brokers.md b/site2/website/versioned_docs/version-2.9.3-deprecated/admin-api-brokers.md index 8bbe0e8f3cda2..f1deabf0b439a 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/admin-api-brokers.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/admin-api-brokers.md @@ -176,7 +176,7 @@ One way to configure a Pulsar [broker](reference-terminology.md#broker) is to su But since all broker configuration in Pulsar is stored in ZooKeeper, configuration values can also be dynamically updated *while the broker is running*. When you update broker configuration dynamically, ZooKeeper will notify the broker of the change and the broker will then override any existing configuration values. -* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. +* The [`brokers`](reference-pulsar-admin.md#brokers) command for the [`pulsar-admin`](reference-pulsar-admin.md) tool has a variety of subcommands that enable you to manipulate a broker's configuration dynamically, enabling you to [update config values](#update-dynamic-configuration) and more. * In the Pulsar admin {@inject: rest:REST:/} API, dynamic configuration is managed through the `/admin/v2/brokers/configuration` endpoint. ### Update dynamic configuration diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/admin-api-clusters.md b/site2/website/versioned_docs/version-2.9.3-deprecated/admin-api-clusters.md index b2f0fa3b25670..1d0c5dc9786f5 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/admin-api-clusters.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/admin-api-clusters.md @@ -89,7 +89,7 @@ When provision a new cluster, you need to initialize that cluster's [metadata](c * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers) that will belong to the cluster. +You must initialize cluster metadata *before* starting up any [brokers](admin-api-brokers.md) that will belong to the cluster. > **No cluster metadata initialization through the REST API or the Java admin API** > @@ -113,11 +113,11 @@ bin/pulsar initialize-cluster-metadata \ ``` -You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication) in your instance. +You'll need to use `--*-tls` flags only if you're using [TLS authentication](security-tls-authentication.md) in your instance. ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing cluster at any time. +You can fetch the [configuration](reference-configuration.md) for an existing cluster at any time. ````mdx-code-block > - For the latest and complete information about `Java admin API`, including classes, methods, descriptions, and more, see [Java admin API doc](https://pulsar.apache.org/api/admin/). -Tenants, like namespaces, can be managed using the [admin API](admin-api-overview). There are currently two configurable aspects of tenants: +Tenants, like namespaces, can be managed using the [admin API](admin-api-overview.md). There are currently two configurable aspects of tenants: * Admin roles * Allowed clusters @@ -118,7 +118,7 @@ admin.tenants().createTenant(tenantName, tenantInfo); ### Get configuration -You can fetch the [configuration](reference-configuration) for an existing tenant at any time. +You can fetch the [configuration](reference-configuration.md) for an existing tenant at any time. ````mdx-code-block To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry). +> To manage message persistence, retention, and expiry in Pulsar, refer to [cookbook](cookbooks-retention-expiry.md). ### Hardware requirements diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-cgo.md b/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-cgo.md index c79f7bb965473..f352f942b7714 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-cgo.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-cgo.md @@ -24,7 +24,7 @@ Currently, the following Go clients are maintained in two repositories. ### Requirements Pulsar Go client library is based on the C++ client library. Follow -the instructions for [C++ library](client-libraries-cpp) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). +the instructions for [C++ library](client-libraries-cpp.md) for installing the binaries through [RPM](client-libraries-cpp.md#rpm), [Deb](client-libraries-cpp.md#deb) or [Homebrew packages](client-libraries-cpp.md#macos). ### Install go package @@ -57,7 +57,7 @@ import "github.com/apache/pulsar/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -75,7 +75,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -236,14 +236,14 @@ Parameter | Description | Default `Topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer will publish messages | `Name` | A name for the producer. If you don't explicitly assign a name, Pulsar will automatically generate a globally unique name that you can access later using the `Name()` method. If you choose to explicitly assign a name, it will need to be unique across *all* Pulsar clusters, otherwise the creation operation will throw an error. | `Properties`| Attach a set of application defined properties to the producer. This properties will be visible in the topic stats | -`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30 seconds +`SendTimeout` | When publishing a message to a topic, the producer will wait for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error will be thrown. If you set `SendTimeout` to -1, the timeout will be set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30 seconds `MaxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `Send` and `SendAsync` methods will fail *unless* `BlockIfQueueFull` is set to `true`. | `MaxPendingMessagesAcrossPartitions` | Set the number of max pending messages across all the partitions. This setting will be used to lower the max pending messages for each partition `MaxPendingMessages(int)`, if the total exceeds the configured value.| `BlockIfQueueFull` | If set to `true`, the producer's `Send` and `SendAsync` methods will block when the outgoing message queue is full rather than failing and throwing an error (the size of that queue is dictated by the `MaxPendingMessages` parameter); if set to `false` (the default), `Send` and `SendAsync` operations will fail and throw a `ProducerQueueIsFullError` when the queue is full. | `false` `MessageRoutingMode` | The message routing logic (for producers on [partitioned topics](concepts-architecture-overview.md#partitioned-topics)). This logic is applied only when no key is set on messages. The available options are: round robin (`pulsar.RoundRobinDistribution`, the default), publishing all messages to a single partition (`pulsar.UseSinglePartition`), or a custom partitioning scheme (`pulsar.CustomPartition`). | `pulsar.RoundRobinDistribution` `HashingScheme` | The hashing function that determines the partition on which a particular message is published (partitioned topics only). The available options are: `pulsar.JavaStringHash` (the equivalent of `String.hashCode()` in Java), `pulsar.Murmur3_32Hash` (applies the [Murmur3](https://en.wikipedia.org/wiki/MurmurHash) hashing function), or `pulsar.BoostHash` (applies the hashing function from C++'s [Boost](https://www.boost.org/doc/libs/1_62_0/doc/html/hash.html) library) | `pulsar.JavaStringHash` `CompressionType` | The message data compression type used by the producer. The available options are [`LZ4`](https://github.com/lz4/lz4), [`ZLIB`](https://zlib.net/), [`ZSTD`](https://facebook.github.io/zstd/) and [`SNAPPY`](https://google.github.io/snappy/). | No compression -`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | +`MessageRouter` | By default, Pulsar uses a round-robin routing scheme for [partitioned topics](cookbooks-partitioned.md). The `MessageRouter` parameter enables you to specify custom routing logic via a function that takes the Pulsar message and topic metadata as an argument and returns an integer (where the ), i.e. a function signature of `func(Message, TopicMetadata) int`. | `Batching` | Control whether automatic batching of messages is enabled for the producer. | false `BatchingMaxPublishDelay` | Set the time period within which the messages sent will be batched (default: 1ms) if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or until | 1ms `BatchingMaxMessages` | Set the maximum number of messages permitted in a batch. (default: 1000) If set to a value greater than 1, messages will be queued until this threshold is reached or batch interval has elapsed | 1000 @@ -512,7 +512,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-cpp.md b/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-cpp.md index 3c94293b7929c..455cf02116d50 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-cpp.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-cpp.md @@ -602,7 +602,7 @@ For complete examples, refer to [C++ client examples](https://github.com/apache/ ## Schema This section describes some examples about schema. For more information about -schema, see [Pulsar schema](schema-get-started). +schema, see [Pulsar schema](schema-get-started.md). ### Avro schema diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-dotnet.md b/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-dotnet.md index ade664cd95277..b574fa0b2e5ed 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-dotnet.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-dotnet.md @@ -170,7 +170,7 @@ var client = PulsarClient.Builder() Currently, the Pulsar C# client supports the TLS (Transport Layer Security) and JWT (JSON Web Token) authentication. -If you have followed [Authentication using TLS](security-tls-authentication), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: +If you have followed [Authentication using TLS](security-tls-authentication.md), you get a certificate and a key. To use them from the Pulsar C# client, follow these steps: 1. Create an unencrypted and password-less pfx file. diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-go.md b/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-go.md index fbb2573aa5db8..6281b03dd8c80 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-go.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-go.md @@ -35,7 +35,7 @@ import "github.com/apache/pulsar-client-go/pulsar" ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here's an example for `localhost`: @@ -61,7 +61,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you're using [TLS](security-tls-authentication) authentication, the URL will look like something like this: +If you're using [TLS](security-tls-authentication.md) authentication, the URL will look like something like this: ```http @@ -844,7 +844,7 @@ Parameter | Description ## TLS encryption and authentication -In order to use [TLS encryption](security-tls-transport), you'll need to configure your client to do so: +In order to use [TLS encryption](security-tls-transport.md), you'll need to configure your client to do so: * Use `pulsar+ssl` URL type * Set `TLSTrustCertsFilePath` to the path to the TLS certs used by your client and the Pulsar broker @@ -864,7 +864,7 @@ opts := pulsar.ClientOptions{ ## OAuth2 authentication -To use [OAuth2 authentication](security-oauth2), you'll need to configure your client to perform the following operations. +To use [OAuth2 authentication](security-oauth2.md), you'll need to configure your client to perform the following operations. This example shows how to configure OAuth2 authentication. ```go diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-java.md b/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-java.md index 613e91f3fe57d..563791618dcd3 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-java.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-java.md @@ -5,7 +5,7 @@ sidebar_label: "Java" original_id: client-libraries-java --- -You can use a Pulsar Java client to create the Java [producer](#producer), [consumer](#consumer), and [readers](#reader) of messages and to perform [administrative tasks](admin-api-overview). The current Java client version is **@pulsar:version@**. +You can use a Pulsar Java client to create the Java [producer](#producer), [consumer](#consumer), and [readers](#reader) of messages and to perform [administrative tasks](admin-api-overview.md). The current Java client version is **@pulsar:version@**. All the methods in [producer](#producer), [consumer](#consumer), and [reader](#reader) of a Java client are thread-safe. @@ -14,10 +14,10 @@ Javadoc for the Pulsar client is divided into two domains by package as follows. Package | Description | Maven Artifact :-------|:------------|:-------------- [`org.apache.pulsar.client.api`](/api/client) | The producer and consumer API | [org.apache.pulsar:pulsar-client:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client%7C@pulsar:version@%7Cjar) -[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) +[`org.apache.pulsar.client.admin`](/api/admin) | The Java [admin API](admin-api-overview.md) | [org.apache.pulsar:pulsar-client-admin:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-admin%7C@pulsar:version@%7Cjar) `org.apache.pulsar.client.all` |Include both `pulsar-client` and `pulsar-client-admin`
    Both `pulsar-client` and `pulsar-client-admin` are shaded packages and they shade dependencies independently. Consequently, the applications using both `pulsar-client` and `pulsar-client-admin` have redundant shaded classes. It would be troublesome if you introduce new dependencies but forget to update shading rules.
    In this case, you can use `pulsar-client-all`, which shades dependencies only one time and reduces the size of dependencies. |[org.apache.pulsar:pulsar-client-all:@pulsar:version@](http://search.maven.org/#artifactdetails%7Corg.apache.pulsar%7Cpulsar-client-all%7C@pulsar:version@%7Cjar) -This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview). +This document focuses only on the client API for producing and consuming messages on Pulsar topics. For how to use the Java admin client, see [Pulsar admin interface](admin-api-overview.md). ## Installation @@ -57,7 +57,7 @@ dependencies { ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. You can assign Pulsar protocol URLs to specific clusters and use the `pulsar` scheme. The default port is `6650`. The following is an example of `localhost`. @@ -83,7 +83,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you use [TLS](security-tls-authentication) authentication, the URL is as follows. +If you use [TLS](security-tls-authentication.md) authentication, the URL is as follows. ```http @@ -114,7 +114,7 @@ PulsarClient client = PulsarClient.builder() ``` > ### Default broker URLs for standalone clusters -> If you run a cluster in [standalone mode](getting-started-standalone), the broker is available at the `pulsar://localhost:6650` URL by default. +> If you run a cluster in [standalone mode](getting-started-standalone.md), the broker is available at the `pulsar://localhost:6650` URL by default. If you create a client, you can use the `loadConf` configuration. The following parameters are available in `loadConf`. @@ -240,7 +240,7 @@ Producer producer = client.newProducer() ### Message routing -When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics cookbook](cookbooks-partitioned). +When using partitioned topics, you can specify the routing mode whenever you publish messages using a producer. For more information on specifying a routing mode using the Java client, see the [Partitioned Topics cookbook](cookbooks-partitioned.md). ### Async send @@ -827,7 +827,7 @@ Total hash range size is 65536, so the max end of the range should be less than ## Schema -In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types). If you construct, say, a [producer](#producer) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. +In Pulsar, all message data consists of byte arrays "under the hood." [Message schemas](schema-get-started.md) enable you to use other types of data when constructing and handling messages (from simple types like strings to more complex, application-specific types.md). If you construct, say, a [producer](#producer) without specifying a schema, then the producer can only produce messages of type `byte[]`. The following is an example. ```java @@ -945,11 +945,11 @@ For example of ProtobufNativeSchema, see [`SchemaDefinition` in `Complex type`]( ## Authentication -Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2). You can use the Pulsar Java client with all of them. +Pulsar currently supports three authentication schemes: [TLS](security-tls-authentication.md), [Athenz](security-athenz.md), and [Oauth2](security-oauth2.md). You can use the Pulsar Java client with all of them. ### TLS Authentication -To use [TLS](security-tls-authentication), you need to set TLS to `true` using the `setUseTls` method, point your Pulsar client to a TLS cert path, and provide paths to cert and key files. +To use [TLS](security-tls-authentication.md), you need to set TLS to `true` using the `setUseTls` method, point your Pulsar client to a TLS cert path, and provide paths to cert and key files. The following is an example. @@ -973,7 +973,7 @@ PulsarClient client = PulsarClient.builder() ### Athenz -To use [Athenz](security-athenz) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: +To use [Athenz](security-athenz.md) as an authentication provider, you need to [use TLS](#tls-authentication) and provide values for four parameters in a hash: * `tenantDomain` * `tenantService` @@ -1011,7 +1011,7 @@ PulsarClient client = PulsarClient.builder() ### Oauth2 -The following example shows how to use [Oauth2](security-oauth2) as an authentication provider for the Pulsar Java client. +The following example shows how to use [Oauth2](security-oauth2.md) as an authentication provider for the Pulsar Java client. You can use the factory method to configure authentication for Pulsar Java client. diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-node.md b/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-node.md index e6584ede93f6d..1ff37b2629466 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-node.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-node.md @@ -48,7 +48,7 @@ Also, this library works only in Node.js 10.x or later because it uses the [`nod ::: ## Connection URLs -To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol) URL. +To connect to Pulsar using client libraries, you need to specify a [Pulsar protocol](developing-binary-protocol.md) URL. Pulsar protocol URLs are assigned to specific clusters, use the `pulsar` scheme and have a default port of 6650. Here is an example for `localhost`: @@ -66,7 +66,7 @@ pulsar://pulsar.us-west.example.com:6650 ``` -If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication), the URL looks like this: +If you are using [TLS encryption](security-tls-transport.md) or [TLS Authentication](security-tls-authentication.md), the URL looks like this: ```http @@ -101,7 +101,7 @@ The following configurable parameters are available for Pulsar clients: | Parameter | Description | Default | | :-------- | :---------- | :------ | | `serviceUrl` | The connection URL for the Pulsar cluster. See [above](#connection-urls) for more info. | | -| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication) for more info. | | +| `authentication` | Configure the authentication provider. (default: no authentication). See [TLS Authentication](security-tls-authentication.md) for more info. | | | `operationTimeoutSeconds` | The timeout for Node.js client operations (creating producers, subscribing to and unsubscribing from [topics](reference-terminology.md#topic)). Retries occur until this threshold is reached, at which point the operation fails. | 30 | | `ioThreads` | The number of threads to use for handling connections to Pulsar [brokers](reference-terminology.md#broker). | 1 | | `messageListenerThreads` | The number of threads used by message listeners ([consumers](#consumers) and [readers](#readers)). | 1 | @@ -154,7 +154,7 @@ Pulsar Node.js producers have the following methods available: | :-------- | :---------- | :------ | | `topic` | The Pulsar [topic](reference-terminology.md#topic) to which the producer publishes messages. | | | `producerName` | A name for the producer. If you do not explicitly assign a name, Pulsar automatically generates a globally unique name. If you choose to explicitly assign a name, it needs to be unique across *all* Pulsar clusters, otherwise the creation operation throws an error. | | -| `sendTimeoutMs` | When publishing a message to a topic, the producer waits for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error is thrown. If you set `sendTimeoutMs` to -1, the timeout is set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication) feature. | 30000 | +| `sendTimeoutMs` | When publishing a message to a topic, the producer waits for an acknowledgment from the responsible Pulsar [broker](reference-terminology.md#broker). If a message is not acknowledged within the threshold set by this parameter, an error is thrown. If you set `sendTimeoutMs` to -1, the timeout is set to infinity (and thus removed). Removing the send timeout is recommended when using Pulsar's [message de-duplication](cookbooks-deduplication.md) feature. | 30000 | | `initialSequenceId` | The initial sequence ID of the message. When producer send message, add sequence ID to message. The ID is increased each time to send. | | | `maxPendingMessages` | The maximum size of the queue holding pending messages (i.e. messages waiting to receive an acknowledgment from the [broker](reference-terminology.md#broker)). By default, when the queue is full all calls to the `send` method fails *unless* `blockIfQueueFull` is set to `true`. | 1000 | | `maxPendingMessagesAcrossPartitions` | The maximum size of the sum of partition's pending queue. | 50000 | diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-python.md b/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-python.md index 72aaf211ae2b2..90cc840daa0a8 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-python.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-python.md @@ -5,7 +5,7 @@ sidebar_label: "Python" original_id: client-libraries-python --- -Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp) and exposes all of the [same features](/api/cpp). You can find the code in the [Python directory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. +Pulsar Python client library is a wrapper over the existing [C++ client library](client-libraries-cpp.md) and exposes all of the [same features](/api/cpp). You can find the code in the [Python directory](https://github.com/apache/pulsar/tree/master/pulsar-client-cpp/python) of the C++ client code. All the methods in producer, consumer, and reader of a Python client are thread-safe. diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-websocket.md b/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-websocket.md index 16eb49af81218..77ec54f803dc5 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-websocket.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/client-libraries-websocket.md @@ -5,14 +5,14 @@ sidebar_label: "WebSocket" original_id: client-libraries-websocket --- -Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. +Pulsar [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) API provides a simple way to interact with Pulsar using languages that do not have an official [client library](getting-started-clients.md). Through WebSocket, you can publish and consume messages and use features available on the [Client Features Matrix](https://github.com/apache/pulsar/wiki/Client-Features-Matrix) page. > You can use Pulsar WebSocket API with any WebSocket client library. See examples for Python and Node.js [below](#client-examples). ## Running the WebSocket service -The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone) already has the WebSocket service enabled. +The standalone variant of Pulsar that we recommend using for [local development](getting-started-standalone.md) already has the WebSocket service enabled. In non-standalone mode, there are two ways to deploy the WebSocket service: diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-architecture-overview.md b/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-architecture-overview.md index 95cf02eac109c..4baa8c30a0d00 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-architecture-overview.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-architecture-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Architecture" original_id: concepts-architecture-overview --- -At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication) data amongst themselves. +At the highest level, a Pulsar instance is composed of one or more Pulsar clusters. Clusters within an instance can [replicate](concepts-replication.md) data amongst themselves. In a Pulsar cluster: @@ -17,20 +17,20 @@ The diagram below provides an illustration of a Pulsar cluster: ![Pulsar architecture diagram](/assets/pulsar-system-architecture.png) -At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication). +At the broader instance level, an instance-wide ZooKeeper cluster called the configuration store handles coordination tasks involving multiple clusters, for example [geo-replication](concepts-replication.md). ## Brokers The Pulsar message broker is a stateless component that's primarily responsible for running two other components: * An HTTP server that exposes a {@inject: rest:REST:/} API for both administrative tasks and [topic lookup](concepts-clients.md#client-setup-phase) for producers and consumers. The producers connect to the brokers to publish messages and the consumers connect to the brokers to consume the messages. -* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol) used for all data transfers +* A dispatcher, which is an asynchronous TCP server over a custom [binary protocol](developing-binary-protocol.md) used for all data transfers Messages are typically dispatched out of a [managed ledger](#managed-ledgers) cache for the sake of performance, *unless* the backlog exceeds the cache size. If the backlog grows too large for the cache, the broker will start reading entries from BookKeeper. -Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java). +Finally, to support geo-replication on global topics, the broker manages replicators that tail the entries published in the local region and republish them to the remote region using the Pulsar [Java client library](client-libraries-java.md). -> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers) guide. +> For a guide to managing Pulsar brokers, see the [brokers](admin-api-brokers.md) guide. ## Clusters @@ -40,9 +40,9 @@ A Pulsar instance consists of one or more Pulsar *clusters*. Clusters, in turn, * A ZooKeeper quorum used for cluster-level configuration and coordination * An ensemble of bookies used for [persistent storage](#persistent-storage) of messages -Clusters can replicate amongst themselves using [geo-replication](concepts-replication). +Clusters can replicate amongst themselves using [geo-replication](concepts-replication.md). -> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters) guide. +> For a guide to managing Pulsar clusters, see the [clusters](admin-api-clusters.md) guide. ## Metadata store @@ -136,17 +136,17 @@ $ bin/pulsar proxy \ ``` > #### Pulsar proxy docs -> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy). +> For documentation on using the Pulsar proxy, see the [Pulsar proxy admin documentation](administration-proxy.md). Some important things to know about the Pulsar proxy: * Connecting clients don't need to provide *any* specific configuration to use the Pulsar proxy. You won't need to update the client configuration for existing applications beyond updating the IP used for the service URL (for example if you're running a load balancer over the Pulsar proxy). -* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication) is supported by the Pulsar proxy +* [TLS encryption](security-tls-transport.md) and [authentication](security-tls-authentication.md) is supported by the Pulsar proxy ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to be able to communicate with an entire Pulsar instance using a single URL. You can use your own service discovery system if you'd like. If you use your own system, there is just one requirement: when a client performs an HTTP request to an endpoint, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to *some* active broker in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. @@ -154,7 +154,7 @@ The diagram below illustrates Pulsar service discovery: ![alt-text](/assets/pulsar-service-discovery.png) -In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python), for example, could access this Pulsar cluster like this: +In this diagram, the Pulsar cluster is addressable via a single DNS name: `pulsar-cluster.acme.com`. A [Python client](client-libraries-python.md), for example, could access this Pulsar cluster like this: ```python diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-authentication.md b/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-authentication.md index b375ecb16a2d8..f6307890c904a 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-authentication.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-authentication.md @@ -5,5 +5,5 @@ sidebar_label: "Authentication and Authorization" original_id: concepts-authentication --- -Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. +Pulsar supports a pluggable [authentication](security-overview.md) mechanism which can be configured at the proxy and/or the broker. Pulsar also supports a pluggable [authorization](security-authorization.md) mechanism. These mechanisms work together to identify the client and its access rights on topics, namespaces and tenants. diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-clients.md b/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-clients.md index b68f76a2d8b08..4040624f7d636 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-clients.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-clients.md @@ -5,12 +5,12 @@ sidebar_label: "Clients" original_id: concepts-clients --- -Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. +Pulsar exposes a client API with language bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md), [C++](client-libraries-cpp.md) and [C#](client-libraries-dotnet.md). The client API optimizes and encapsulates Pulsar's client-broker communication protocol and exposes a simple and intuitive API for use by applications. Under the hood, the current official Pulsar client libraries support transparent reconnection and/or connection failover to brokers, queuing of messages until acknowledged by the broker, and heuristics such as connection retries with backoff. > **Custom client libraries** -> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol). +> If you'd like to create your own client library, we recommend consulting the documentation on Pulsar's custom [binary protocol](developing-binary-protocol.md). ## Client setup phase @@ -38,7 +38,7 @@ Internally, the reader interface is implemented as a consumer using an exclusive [ **IMPORTANT** ] -Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. +Unlike subscription/consumer, readers are non-durable in nature and does not prevent data in a topic from being deleted, thus it is ***strongly*** advised that [data retention](cookbooks-retention-expiry.md) be configured. If data retention for a topic is not configured for an adequate amount of time, messages that the reader has not yet read might be deleted . This causes the readers to essentially skip messages. Configuring the data retention for a topic guarantees the reader with a certain duration to read a message. Please also note that a reader can have a "backlog", but the metric is only used for users to know how behind the reader is. The metric is not considered for any backlog quota calculations. diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-messaging.md b/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-messaging.md index da65d508392d2..c7469606f3055 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-messaging.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-messaging.md @@ -23,8 +23,8 @@ Messages are the basic "unit" of Pulsar. The following table lists the component Component | Description :---------|:------- -Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started). -Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction). +Value / data payload | The data carried by the message. All Pulsar messages contain raw bytes, although message data can also conform to data [schemas](schema-get-started.md). +Key | Messages are optionally tagged with keys, which is useful for things like [topic compaction](concepts-topic-compaction.md). Properties | An optional key/value map of user-defined properties. Producer name | The name of the producer who produces the message. If you do not specify a producer name, the default name is used. Sequence ID | Each Pulsar message belongs to an ordered sequence on its topic. The sequence ID of the message is its order in that sequence. @@ -52,7 +52,7 @@ The default size of a message is 5 MB. You can configure the max size of a messa ``` -> For more information on Pulsar messages, see Pulsar [binary protocol](developing-binary-protocol). +> For more information on Pulsar messages, see Pulsar [binary protocol](developing-binary-protocol.md). ## Producers @@ -149,7 +149,7 @@ Messages are received from [brokers](reference-terminology.md#broker) either syn ### Listeners -Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. +Client libraries provide listener implementation for consumers. For example, the [Java client](client-libraries-java.md) provides a {@inject: javadoc:MesssageListener:/client/org/apache/pulsar/client/api/MessageListener} interface. In this interface, the `received` method is called whenever a new message is received. ### Acknowledgement @@ -502,7 +502,7 @@ Decisions about routing and subscription modes can be made separately in most ca There is no difference between partitioned topics and normal topics in terms of how subscription modes work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer. -Partitioned topics need to be explicitly created via the [admin API](admin-api-overview). The number of partitions can be specified when creating the topic. +Partitioned topics need to be explicitly created via the [admin API](admin-api-overview.md). The number of partitions can be specified when creating the topic. ### Routing modes @@ -514,7 +514,7 @@ Mode | Description :--------|:------------ `RoundRobinPartition` | If no key is provided, the producer will publish messages across all partitions in round-robin fashion to achieve maximum throughput. Please note that round-robin is not done per individual message but rather it's set to the same boundary of batching delay, to ensure batching is effective. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. This is the default mode. `SinglePartition` | If no key is provided, the producer will randomly pick one single partition and publish all the messages into that partition. While if a key is specified on the message, the partitioned producer will hash the key and assign message to a particular partition. -`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. +`CustomPartition` | Use custom message router implementation that will be called to determine the partition for a particular message. User can create a custom routing mode by using the [Java client](client-libraries-java.md) and implementing the {@inject: javadoc:MessageRouter:/client/org/apache/pulsar/client/api/MessageRouter} interface. ### Ordering guarantee @@ -552,7 +552,7 @@ non-persistent://tenant/namespace/topic ``` -> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent). +> For more info on using non-persistent topics, see the [Non-persistent messaging cookbook](cookbooks-non-persistent.md). In non-persistent topics, brokers immediately deliver messages to all connected subscribers *without persisting them* in [BookKeeper](concepts-architecture-overview.md#persistent-storage). If a subscriber is disconnected, the broker will not be able to deliver those in-transit messages, and subscribers will never be able to receive those messages again. Eliminating the persistent storage step makes messaging on non-persistent topics slightly faster than on persistent topics in some cases, but with the caveat that some of the core benefits of Pulsar are lost. @@ -638,7 +638,7 @@ Pulsar has two features, however, that enable you to override this default behav * Message **retention** enables you to store messages that have been acknowledged by a consumer * Message **expiry** enables you to set a time to live (TTL) for messages that have not yet been acknowledged -> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry) cookbook. +> All message retention and expiry is managed at the [namespace](#namespaces) level. For a how-to, see the [Message retention and expiry](cookbooks-retention-expiry.md) cookbook. The diagram below illustrates both concepts: @@ -661,12 +661,12 @@ Message deduplication is disabled in the scenario shown at the top. Here, a prod In the second scenario at the bottom, the producer publishes message 1, which is received by the broker and persisted, as in the first scenario. When the producer attempts to publish the message again, however, the broker knows that it has already seen message 1 and thus does not persist the message. -> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication). +> Message deduplication is handled at the namespace level or the topic level. For more instructions, see the [message deduplication cookbook](cookbooks-deduplication.md). ### Producer idempotency -The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication). +The other available approach to message deduplication is to ensure that each message is *only produced once*. This approach is typically called **producer idempotency**. The drawback of this approach is that it defers the work of message deduplication to the application. In Pulsar, this is handled at the [broker](reference-terminology.md#broker) level, so you do not need to modify your Pulsar client code. Instead, you only need to make administrative changes. For details, see [Managing message deduplication](cookbooks-deduplication.md). ### Deduplication and effectively-once semantics diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-multi-tenancy.md b/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-multi-tenancy.md index 730458c904618..93a59557b2efc 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-multi-tenancy.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-multi-tenancy.md @@ -5,7 +5,7 @@ sidebar_label: "Multi Tenancy" original_id: concepts-multi-tenancy --- -Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. +Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own [authentication and authorization](security-overview.md) scheme applied to them. They are also the administrative unit at which storage quotas, [message TTL](cookbooks-retention-expiry.md#time-to-live-ttl), and isolation policies can be managed. The multi-tenant nature of Pulsar is reflected mostly visibly in topic URLs, which have this structure: @@ -21,7 +21,7 @@ As you can see, the tenant is the most basic unit of categorization for topics ( To each tenant in a Pulsar instance you can assign: -* An [authorization](security-authorization) scheme +* An [authorization](security-authorization.md) scheme * The set of [clusters](reference-terminology.md#cluster) to which the tenant's configuration applies ## Namespaces @@ -29,7 +29,7 @@ To each tenant in a Pulsar instance you can assign: Tenants and namespaces are two key concepts of Pulsar to support multi-tenancy. * Pulsar is provisioned for specified tenants with appropriate capacity allocated to the tenant. -* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. +* A namespace is the administrative unit nomenclature within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. A tenant may create multiple namespaces via self-administration using the REST API and the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. For instance, a tenant with different applications can create a separate namespace for each application. Names for topics in the same namespace will look like this: diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-overview.md b/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-overview.md index 310a78dff4565..c643aa0ce7bbc 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-overview.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-overview.md @@ -9,15 +9,15 @@ Pulsar is a multi-tenant, high-performance solution for server-to-server messagi Key features of Pulsar are listed below: -* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo) of messages across clusters. +* Native support for multiple clusters in a Pulsar instance, with seamless [geo-replication](administration-geo.md) of messages across clusters. * Very low publish and end-to-end latency. * Seamless scalability to over a million topics. -* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp). +* A simple [client API](concepts-clients.md) with bindings for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) and [C++](client-libraries-cpp.md). * Multiple [subscription modes](concepts-messaging.md#subscription-modes) ([exclusive](concepts-messaging.md#exclusive), [shared](concepts-messaging.md#shared), and [failover](concepts-messaging.md#failover)) for topics. * Guaranteed message delivery with [persistent message storage](concepts-architecture-overview.md#persistent-storage) provided by [Apache BookKeeper](http://bookkeeper.apache.org/). -* A serverless light-weight computing framework [Pulsar Functions](functions-overview) offers the capability for stream-native data processing. -* A serverless connector framework [Pulsar IO](io-overview), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. -* [Tiered Storage](concepts-tiered-storage) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. +* A serverless light-weight computing framework [Pulsar Functions](functions-overview.md) offers the capability for stream-native data processing. +* A serverless connector framework [Pulsar IO](io-overview.md), which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar. +* [Tiered Storage](concepts-tiered-storage.md) offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out. ## Contents diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-replication.md b/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-replication.md index 6e23962e99771..799f0eb4d92c6 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-replication.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-replication.md @@ -5,5 +5,5 @@ sidebar_label: "Geo Replication" original_id: concepts-replication --- -Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo) in Pulsar enables you to do that. +Pulsar enables messages to be produced and consumed in different geo-locations. For instance, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. [Geo-replication](administration-geo.md) in Pulsar enables you to do that. diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-tiered-storage.md b/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-tiered-storage.md index 0b45b0a2ed501..f6988e53a8cd4 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-tiered-storage.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-tiered-storage.md @@ -15,4 +15,4 @@ One way to alleviate this cost is to use Tiered Storage. With tiered storage, ol Pulsar currently supports S3, Google Cloud Storage (GCS), and filesystem for [long term store](https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/). Offloading to long term storage triggered via a Rest API or command line interface. The user passes in the amount of topic data they wish to retain on BookKeeper, and the broker will copy the backlog data to long term storage. The original data will then be deleted from BookKeeper after a configured delay (4 hours by default). -> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage). +> For a guide for setting up tiered storage, see the [Tiered storage cookbook](cookbooks-tiered-storage.md). diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-topic-compaction.md b/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-topic-compaction.md index e402c965384fd..34b7ed7fbbd31 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-topic-compaction.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/concepts-topic-compaction.md @@ -7,7 +7,7 @@ original_id: concepts-topic-compaction Pulsar was built with highly scalable [persistent storage](concepts-architecture-overview.md#persistent-storage) of message data as a primary objective. Pulsar topics enable you to persistently store as many unacknowledged messages as you need while preserving message ordering. By default, Pulsar stores *all* unacknowledged/unprocessed messages produced on a topic. Accumulating many unacknowledged messages on a topic is necessary for many Pulsar use cases but it can also be very time intensive for Pulsar consumers to "rewind" through the entire log of messages. -> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction). +> For a more practical guide to topic compaction, see the [Topic compaction cookbook](cookbooks-compaction.md). For some use cases consumers don't need a complete "image" of the topic log. They may only need a few values to construct a more "shallow" image of the log, perhaps even just the most recent value. For these kinds of use cases Pulsar offers **topic compaction**. When you run compaction on a topic, Pulsar goes through a topic's backlog and removes messages that are *obscured* by later messages, i.e. it goes through the topic on a per-key basis and leaves only the most recent message associated with that key. @@ -24,7 +24,7 @@ Pulsar's topic compaction feature: ## How topic compaction works -When topic compaction is triggered [via the CLI](cookbooks-compaction), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. +When topic compaction is triggered [via the CLI](cookbooks-compaction.md), Pulsar will iterate over the entire topic from beginning to end. For each key that it encounters the compaction routine will keep a record of the latest occurrence of that key. After that, the broker will create a new [BookKeeper ledger](concepts-architecture-overview.md#ledgers) and make a second iteration through each message on the topic. For each message, if the key matches the latest occurrence of that key, then the key's data payload, message ID, and metadata will be written to the newly created ledger. If the key doesn't match the latest then the message will be skipped and left alone. If any given message has an empty payload, it will be skipped and considered deleted (akin to the concept of [tombstones](https://en.wikipedia.org/wiki/Tombstone_(data_store)) in key-value databases). At the end of this second iteration through the topic, the newly created BookKeeper ledger is closed and two things are written to the topic's metadata: the ID of the BookKeeper ledger and the message ID of the last compacted message (this is known as the **compaction horizon** of the topic). Once this metadata is written compaction is complete. diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/cookbooks-compaction.md b/site2/website/versioned_docs/version-2.9.3-deprecated/cookbooks-compaction.md index 6fcbc197f9ecb..dfa314727241a 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/cookbooks-compaction.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/cookbooks-compaction.md @@ -45,7 +45,7 @@ Configuring the compaction threshold on a namespace will apply to all topics wit ## Triggering compaction manually -In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin) CLI tool. Here's an example: +In order to run compaction on a topic, you need to use the [`topics compact`](reference-pulsar-admin.md#topics-compact) command for the [`pulsar-admin`](reference-pulsar-admin.md) CLI tool. Here's an example: ```bash diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/cookbooks-non-persistent.md b/site2/website/versioned_docs/version-2.9.3-deprecated/cookbooks-non-persistent.md index 391569a8cbc6b..178301e86eb8d 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/cookbooks-non-persistent.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/cookbooks-non-persistent.md @@ -41,7 +41,7 @@ $ bin/pulsar-client produce non-persistent://public/default/example-np-topic \ ``` -> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics) guide. +> For a more thorough guide to non-persistent topics from an administrative perspective, see the [Non-persistent topics](admin-api-topics.md) guide. ## Enabling @@ -55,7 +55,7 @@ If you'd like to enable *only* non-persistent topics in a broker, you can set th ## Managing with cli -Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin) non-persistent topics under a namespace, and more. +Non-persistent topics can be managed using the [`pulsar-admin non-persistent`](reference-pulsar-admin.md#non-persistent) command-line interface. With that interface you can perform actions like [create a partitioned non-persistent topic](reference-pulsar-admin.md#non-persistent-create-partitioned-topic), get [stats](reference-pulsar-admin.md#non-persistent-stats) for a non-persistent topic, [list](reference-pulsar-admin.md) non-persistent topics under a namespace, and more. ## Using with Pulsar clients diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/cookbooks-partitioned.md b/site2/website/versioned_docs/version-2.9.3-deprecated/cookbooks-partitioned.md index 7882fb9025312..fb9ac354cc6d6 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/cookbooks-partitioned.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/cookbooks-partitioned.md @@ -4,4 +4,4 @@ title: Partitioned topics sidebar_label: "Partitioned Topics" original_id: cookbooks-partitioned --- -For details of the content, refer to [manage topics](admin-api-topics). \ No newline at end of file +For details of the content, refer to [manage topics](admin-api-topics.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/cookbooks-retention-expiry.md b/site2/website/versioned_docs/version-2.9.3-deprecated/cookbooks-retention-expiry.md index 0283a64312ca7..c8c46b3caa1be 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/cookbooks-retention-expiry.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/cookbooks-retention-expiry.md @@ -22,7 +22,7 @@ In Pulsar, you can modify this behavior, with namespace granularity, in two ways * You can persistently store messages that are not within a backlog (because they've been acknowledged by on every existing subscription, or because there are no subscriptions) by setting [retention policies](#retention-policies). * Messages that are not acknowledged within a specified timeframe can be automatically acknowledged, by specifying the [time to live](#time-to-live-ttl) (TTL). -Pulsar's [admin interface](admin-api-overview) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). +Pulsar's [admin interface](admin-api-overview.md) enables you to manage both retention policies and TTL with namespace granularity (and thus within a specific tenant and either on a specific cluster or in the [`global`](concepts-architecture-overview.md#global-cluster) cluster). > #### Retention and TTL solve two different problems diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-aws.md b/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-aws.md index 7ae3bb0fbb8b7..0d94fc13cdb2d 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-aws.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-aws.md @@ -5,7 +5,7 @@ sidebar_label: "Amazon Web Services" original_id: deploy-aws --- -> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster). +> For instructions on deploying a single Pulsar cluster manually rather than using Terraform and Ansible, see [Deploying a Pulsar cluster on bare metal](deploy-bare-metal.md). For instructions on manually deploying a multi-cluster Pulsar instance, see [Deploying a Pulsar instance on bare metal](deploy-bare-metal-multi-cluster.md). One of the easiest ways to get a Pulsar [cluster](reference-terminology.md#cluster) running on [Amazon Web Services](https://aws.amazon.com/) (AWS) is to use the [Terraform](https://terraform.io) infrastructure provisioning tool and the [Ansible](https://www.ansible.com) server automation tool. Terraform can create the resources necessary for running the Pulsar cluster---[EC2](https://aws.amazon.com/ec2/) instances, networking and security infrastructure, etc.---While Ansible can install and run Pulsar on the provisioned resources. @@ -210,7 +210,7 @@ Remember to enter this command just only once. If you attempt to enter this comm Once you have created the necessary AWS resources using Terraform, you can install and run Pulsar on the Terraform-created EC2 instances using Ansible. -(Optional) If you want to use any [built-in IO connectors](io-connectors) , edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. +(Optional) If you want to use any [built-in IO connectors](io-connectors.md) , edit the `Download Pulsar IO packages` task in the `deploy-pulsar.yaml` file and uncomment the connectors you want to use. To run the playbook, enter this command: diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-bare-metal-multi-cluster.md b/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-bare-metal-multi-cluster.md index 1af30a644abe8..49fd3938c64d4 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-bare-metal-multi-cluster.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-bare-metal-multi-cluster.md @@ -7,13 +7,13 @@ original_id: deploy-bare-metal-multi-cluster :::tip -1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster). -2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster.md). +2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: -A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo).Deploying a multi-cluster Pulsar instance consists of the following steps: +A Pulsar instance consists of multiple Pulsar clusters working in unison. You can distribute clusters across data centers or geographical regions and replicate the clusters amongst themselves using [geo-replication](administration-geo.md).Deploying a multi-cluster Pulsar instance consists of the following steps: 1. Deploying two separate ZooKeeper quorums: a local quorum for each cluster in the instance and a configuration store quorum for instance-wide tasks 2. Initializing cluster metadata for each cluster @@ -22,7 +22,7 @@ A Pulsar instance consists of multiple Pulsar clusters working in unison. You ca > #### Run Pulsar locally or on Kubernetes? -> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. +> This guide shows you how to deploy Pulsar in production in a non-Kubernetes environment. If you want to run a standalone Pulsar cluster on a single machine for development purposes, see the [Setting up a local cluster](getting-started-standalone.md) guide. If you want to run Pulsar on [Kubernetes](https://kubernetes.io), see the [Pulsar on Kubernetes](deploy-kubernetes.md) guide, which includes sections on running Pulsar on Kubernetes, on Google Kubernetes Engine and on Amazon Web Services. ## System requirement @@ -65,7 +65,7 @@ The Pulsar binary package initially contains the following directories: Directory | Contains :---------|:-------- -`bin` | [Command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` | [Command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `examples` | A Java JAR file containing example [Pulsar Functions](functions-overview.md) `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -244,7 +244,7 @@ As you can see from the example above, you need to specify the following: * The web service URL for the cluster * A broker service URL enabling interaction with the [brokers](reference-terminology.md#broker) in the cluster -If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. +If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. Make sure to run `initialize-cluster-metadata` for each cluster in your instance. @@ -363,16 +363,16 @@ $ bin/pulsar broker ## Service discovery -[Clients](getting-started-clients) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. +[Clients](getting-started-clients.md) connecting to Pulsar brokers need to communicate with an entire Pulsar instance using a single URL. -You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. +You can use your own service discovery system. If you use your own system, you only need to satisfy just one requirement: when a client performs an HTTP request to an [endpoint](reference-configuration.md) for a Pulsar cluster, such as `http://pulsar.us-west.example.com:8080`, the client needs to be redirected to some active brokers in the desired cluster, whether via DNS, an HTTP or IP redirect, or some other means. > **Service discovery already provided by many scheduling systems** -> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. +> Many large-scale deployment systems, such as [Kubernetes](deploy-kubernetes.md), have service discovery systems built in. If you run Pulsar on such a system, you may not need to provide your own service discovery mechanism. ## Admin client and verification -At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. +At this point your Pulsar instance should be ready to use. You can now configure client machines that can serve as [administrative clients](admin-api-overview.md) for each cluster. You can use the [`conf/client.conf`](reference-configuration.md#client) configuration file to configure admin clients. The most important thing is that you point the [`serviceUrl`](reference-configuration.md#client-serviceUrl) parameter to the correct service URL for the cluster: diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-bare-metal.md b/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-bare-metal.md index 6761a7c8fb110..292157e3ddc89 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-bare-metal.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-bare-metal.md @@ -7,9 +7,9 @@ original_id: deploy-bare-metal :::tip -1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster). -2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview). -3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage). +1. You can use single-cluster Pulsar installation in most use cases, such as experimenting with Pulsar or using Pulsar in a startup or in a single team. If you need to run a multi-cluster Pulsar instance, see the [guide](deploy-bare-metal-multi-cluster.md). +2. If you want to use all built-in [Pulsar IO](io-overview.md) connectors, you need to download `apache-pulsar-io-connectors`package and install `apache-pulsar-io-connectors` under `connectors` directory in the pulsar directory on every broker node or on every function-worker node if you have run a separate cluster of function workers for [Pulsar Functions](functions-overview.md). +3. If you want to use [Tiered Storage](concepts-tiered-storage.md) feature in your Pulsar deployment, you need to download `apache-pulsar-offloaders`package and install `apache-pulsar-offloaders` under `offloaders` directory in the Pulsar directory on every broker node. For more details of how to configure this feature, you can refer to the [Tiered storage cookbook](cookbooks-tiered-storage.md). ::: @@ -126,7 +126,7 @@ The extracted directory contains the following subdirectories: Directory | Contains :---------|:-------- -`bin` |[command-line tools](reference-cli-tools) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) +`bin` |[command-line tools](reference-cli-tools.md) of Pulsar, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/) `conf` | Configuration files for Pulsar, including for [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more `data` | The data storage directory that ZooKeeper and BookKeeper use `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files that Pulsar uses @@ -279,9 +279,9 @@ Flag | Description `--zookeeper` | A "local" ZooKeeper connection string for the cluster. This connection string only needs to include *one* machine in the ZooKeeper cluster. `--configuration-store` | The configuration store connection string for the entire instance. As with the `--zookeeper` flag, this connection string only needs to include *one* machine in the ZooKeeper cluster. `--web-service-url` | The web service URL for the cluster, plus a port. This URL should be a standard DNS name. The default port is 8080 (you had better not use a different port). -`--web-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). +`--web-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster. The default port is 8443 (you had better not use a different port). `--broker-service-url` | A broker service URL enabling interaction with the brokers in the cluster. This URL should not use the same DNS name as the web service URL but should use the `pulsar` scheme instead. The default port is 6650 (you had better not use a different port). -`--broker-service-url-tls` | If you use [TLS](security-tls-transport), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). +`--broker-service-url-tls` | If you use [TLS](security-tls-transport.md), you also need to specify a TLS web service URL for the cluster as well as a TLS broker service URL for the brokers in the cluster. The default port is 6651 (you had better not use a different port). > If you do not have a DNS server, you can use multi-host format in the service URL with the following settings: @@ -421,7 +421,7 @@ webServicePortTls=8443 ### Enable Pulsar Functions (optional) -If you want to enable [Pulsar Functions](functions-overview), you can follow the instructions as below: +If you want to enable [Pulsar Functions](functions-overview.md), you can follow the instructions as below: 1. Edit `conf/broker.conf` to enable functions worker, by setting `functionsWorkerEnabled` to `true`. @@ -439,7 +439,7 @@ If you want to enable [Pulsar Functions](functions-overview), you can follow the ``` -If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker). +If you want to learn more options about deploying the functions worker, check out [Deploy and manage functions worker](functions-worker.md). ### Start Brokers diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-dcos.md b/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-dcos.md index 49cf624ea24f1..35a0a83d716ad 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-dcos.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-dcos.md @@ -7,7 +7,7 @@ original_id: deploy-dcos :::tip -To enable all built-in [Pulsar IO](io-overview) connectors in your Pulsar deployment, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). +To enable all built-in [Pulsar IO](io-overview.md) connectors in your Pulsar deployment, we recommend you use `apachepulsar/pulsar-all` image instead of `apachepulsar/pulsar` image; the former has already bundled [all built-in connectors](io-overview.md#working-with-connectors). ::: diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-docker.md b/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-docker.md index 64a0939aa787b..8348d78deb237 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-docker.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-docker.md @@ -57,4 +57,4 @@ docker network connect pulsar broker To check whether the containers are successfully connected to the network, enter the `docker network inspect pulsar` command. -For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal). +For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal.md). diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-kubernetes.md b/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-kubernetes.md index dc7123d000467..1aefc6ad79f71 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-kubernetes.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-kubernetes.md @@ -6,6 +6,6 @@ original_id: deploy-kubernetes --- To get up and running with these charts as fast as possible, in a **non-production** use case, we provide -a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install). \ No newline at end of file +To configure and install a Pulsar cluster on Kubernetes for production usage, follow the complete [Installation Guide](helm-install.md). \ No newline at end of file diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-monitoring.md b/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-monitoring.md index 87d435050df27..2b5c19344dc8c 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-monitoring.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/deploy-monitoring.md @@ -113,7 +113,7 @@ http://$FUNCTIONS_WORKER_ADDRESS:$WORKER_PORT/metrics: You can use Prometheus to collect all the metrics exposed for Pulsar components and set up [Grafana](https://grafana.com/) dashboards to display the metrics and monitor your Pulsar cluster. For details, refer to [Prometheus guide](https://prometheus.io/docs/introduction/getting_started/). -When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy). +When you run Pulsar on bare metal, you can provide the list of nodes to be probed. When you deploy Pulsar in a Kubernetes cluster, the monitoring is setup automatically. For details, refer to [Kubernetes instructions](helm-deploy.md). ## Dashboards @@ -121,7 +121,7 @@ When you collect time series statistics, the major problem is to make sure the n ### Pulsar per-topic dashboard -The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager). +The per-topic dashboard instructions are available at [Pulsar manager](administration-pulsar-manager.md). ### Grafana diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/develop-schema.md b/site2/website/versioned_docs/version-2.9.3-deprecated/develop-schema.md index e71c04ef60dc1..2d4461a5ea2b5 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/develop-schema.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/develop-schema.md @@ -5,7 +5,7 @@ sidebar_label: "Custom schema storage" original_id: develop-schema --- -By default, Pulsar stores data type [schemas](concepts-schema-registry) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. +By default, Pulsar stores data type [schemas](concepts-schema-registry.md) in [Apache BookKeeper](https://bookkeeper.apache.org) (which is deployed alongside Pulsar). You can, however, use another storage system if you wish. This doc walks you through creating your own schema storage implementation. In order to use a non-default (i.e. non-BookKeeper) storage system for Pulsar schemas, you need to implement two Java interfaces: [`SchemaStorage`](#schemastorage-interface) and [`SchemaStorageFactory`](#schemastoragefactory-interface). diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/functions-deploy.md b/site2/website/versioned_docs/version-2.9.3-deprecated/functions-deploy.md index d8fe0fc14e5bb..2a0d68d6c623c 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/functions-deploy.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/functions-deploy.md @@ -9,12 +9,12 @@ original_id: functions-deploy To deploy and manage Pulsar Functions, you need to have a Pulsar cluster running. There are several options for this: -* You can run a [standalone cluster](getting-started-standalone) locally on your own machine. -* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal), [DC/OS](https://dcos.io/), and more. +* You can run a [standalone cluster](getting-started-standalone.md) locally on your own machine. +* You can deploy a Pulsar cluster on [Kubernetes](deploy-kubernetes.md), [Amazon Web Services](deploy-aws.md), [bare metal](deploy-bare-metal.md), [DC/OS](https://dcos.io/), and more. If you run a non-[standalone](reference-terminology.md#standalone) cluster, you need to obtain the service URL for the cluster. How you obtain the service URL depends on how you deploy your Pulsar cluster. -If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker). +If you want to deploy and trigger Python user-defined functions, you need to install [the pulsar python client](http://pulsar.apache.org/docs/en/client-libraries-python/) on all the machines running [functions workers](functions-worker.md). ## Command-line interface @@ -179,7 +179,7 @@ $ bin/pulsar-admin functions create \ Package management enables version management and simplifies the upgrade and rollback processes for Functions, Sinks, and Sources. When you use the same function, sink and source in different namespaces, you can upload them to a common package management system. -To use [Package management service](admin-api-packages), ensure that the package management service has been enabled in your cluster by setting the following properties in `broker.conf`. +To use [Package management service](admin-api-packages.md), ensure that the package management service has been enabled in your cluster by setting the following properties in `broker.conf`. > Note: Package management service is not enabled by default. diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/functions-develop.md b/site2/website/versioned_docs/version-2.9.3-deprecated/functions-develop.md index 36ea4b4a21073..2e29aa1c47400 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/functions-develop.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/functions-develop.md @@ -1144,7 +1144,7 @@ You can monitor Pulsar Functions that have been deployed with the following meth - Check the metrics provided by Pulsar. - Pulsar Functions expose the metrics that can be collected and used for monitoring the health of **Java, Python, and Go** functions. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar Functions expose the metrics that can be collected and used for monitoring the health of **Java, Python, and Go** functions. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. For the complete list of the function metrics, see [here](reference-metrics.md#pulsar-functions). @@ -1214,7 +1214,7 @@ Currently, the feature is not available in Go. ## Security -If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker). For more details, refer to [Security settings](functions-worker.md#security-settings). +If you want to enable security on Pulsar Functions, first you should enable security on [Functions Workers](functions-worker.md). For more details, refer to [Security settings](functions-worker.md#security-settings). Pulsar Functions can support the following providers: diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/functions-overview.md b/site2/website/versioned_docs/version-2.9.3-deprecated/functions-overview.md index 25dc6029acf70..816d301e0fd0e 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/functions-overview.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/functions-overview.md @@ -164,7 +164,7 @@ tenant/namespace/name FQFNs enable you to create multiple functions with the same name provided that they are in different namespaces. ## Supported languages -Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop). +Currently, you can write Pulsar Functions in Java, Python, and Go. For details, refer to [Develop Pulsar Functions](functions-develop.md). ## Processing guarantees Pulsar Functions provide three different messaging semantics that you can apply to any function. diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/functions-package.md b/site2/website/versioned_docs/version-2.9.3-deprecated/functions-package.md index 2666d5944b2e0..db2c4e987dc7b 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/functions-package.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/functions-package.md @@ -15,7 +15,7 @@ Currently, the window function is not available in Python and Go. ## Prerequisite -Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm). +Before running a Pulsar function, you need to start Pulsar. You can [run a standalone Pulsar in Docker](getting-started-docker.md), or [run Pulsar in Kubernetes](getting-started-helm.md). To check whether the Docker image starts, you can use the `docker ps` command. diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/functions-worker.md b/site2/website/versioned_docs/version-2.9.3-deprecated/functions-worker.md index d90b3456e6953..35e26926bb7ab 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/functions-worker.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/functions-worker.md @@ -4,7 +4,7 @@ title: Deploy and manage functions worker sidebar_label: "Setup: Pulsar Functions Worker" original_id: functions-worker --- -Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime). +Before using Pulsar Functions, you need to learn how to set up Pulsar Functions worker and how to [configure Functions runtime](functions-runtime.md). Pulsar `functions-worker` is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either based on your requirements. - [run with brokers](#run-functions-worker-with-brokers) @@ -178,7 +178,7 @@ brokerClientTrustCertsFilePath: /path/to/ca.cert.pem ``` -For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport). +For details on TLS encryption, refer to [Transport Encryption using TLS](security-tls-transport.md). ##### Enable Authentication Provider @@ -198,7 +198,7 @@ authenticationProviders: [ provider1, provider2 ] ``` For *TLS Authentication* provider, follow the example below to add the necessary settings. -See [TLS Authentication](security-tls-authentication) for more details. +See [TLS Authentication](security-tls-authentication.md) for more details. ``` @@ -222,7 +222,7 @@ properties: ``` For *Token Authentication* provider, add necessary settings for `properties` if needed. -See [Token Authentication](security-jwt) for more details. +See [Token Authentication](security-jwt.md) for more details. Note: key files must be DER-encoded ``` @@ -261,7 +261,7 @@ superUserRoles: You can use the public and private key pair that the application configures to perform encryption. Only the consumers with a valid key can decrypt the encrypted messages. -To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption). +To enable End-to-End encryption on Functions Worker, you can set it by specifying `--producer-config` in the command line terminal, for more information, please refer to [here](security-encryption.md). We include the relevant configuration information of `CryptoConfig` into `ProducerConfig`. The specific configurable field information about `CryptoConfig` is as follows: diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/getting-started-docker.md b/site2/website/versioned_docs/version-2.9.3-deprecated/getting-started-docker.md index 71f1bb9734fb7..de5ead69e164b 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/getting-started-docker.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/getting-started-docker.md @@ -43,8 +43,8 @@ When you start a local standalone cluster, a `public/default` namespace is creat ## Use Pulsar in Docker -Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python) -and [C++](client-libraries-cpp). If you're running a local standalone cluster, you can +Pulsar offers client libraries for [Java](client-libraries-java.md), [Go](client-libraries-go.md), [Python](client-libraries-python.md) +and [C++](client-libraries-cpp.md). If you're running a local standalone cluster, you can use one of these root URLs to interact with your cluster: * `pulsar://localhost:6650` @@ -103,7 +103,7 @@ client.close() ## Get the topic statistics In Pulsar, you can use REST, Java, or command-line tools to control every aspect of the system. -For details on APIs, refer to [Admin API Overview](admin-api-overview). +For details on APIs, refer to [Admin API Overview](admin-api-overview.md). In the simplest example, you can use curl to probe the stats for a particular topic: diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/getting-started-helm.md b/site2/website/versioned_docs/version-2.9.3-deprecated/getting-started-helm.md index 98a3097d436db..5e9f7044a6d74 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/getting-started-helm.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/getting-started-helm.md @@ -13,7 +13,7 @@ This section guides you through every step of installing and running Apache Puls - Produce and consume messages using Pulsar clients - Monitor Apache Pulsar status with Prometheus and Grafana -For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy). +For deploying a Pulsar cluster for production usage, read the documentation on [how to configure and install a Pulsar Helm chart](helm-deploy.md). ## Prerequisite @@ -29,7 +29,7 @@ For the following steps, step 2 and step 3 are for **developers** and step 4 and ## Step 0: Prepare a Kubernetes cluster -Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare) to prepare a Kubernetes cluster. +Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow [the instructions](helm-prepare.md) to prepare a Kubernetes cluster. We use [Minikube](https://minikube.sigs.k8s.io/docs/start/) in this quick start guide. To prepare a Kubernetes cluster, follow these steps: @@ -409,7 +409,7 @@ Then you can proceed with the following steps: ## Step 4: Use Pulsar Manager to manage the cluster -[Pulsar Manager](administration-pulsar-manager) is a web-based GUI management tool for managing and monitoring Pulsar. +[Pulsar Manager](administration-pulsar-manager.md) is a web-based GUI management tool for managing and monitoring Pulsar. 1. By default, the `Pulsar Manager` is exposed as a separate `LoadBalancer`. You can open the Pulsar Manager UI using the following command: diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/getting-started-pulsar.md b/site2/website/versioned_docs/version-2.9.3-deprecated/getting-started-pulsar.md index 11c5e66cb62c2..752590f57b558 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/getting-started-pulsar.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/getting-started-pulsar.md @@ -5,13 +5,13 @@ sidebar_label: "Pulsar 2.0" original_id: pulsar-2.0 --- -Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview) feature, some terminology changes, and more. +Pulsar 2.0 is a major new release for Pulsar that brings some bold changes to the platform, including [simplified topic names](#topic-names), the addition of the [Pulsar Functions](functions-overview.md) feature, some terminology changes, and more. ## New features in Pulsar 2.0 Feature | Description :-------|:----------- -[Pulsar Functions](functions-overview) | A lightweight compute option for Pulsar +[Pulsar Functions](functions-overview.md) | A lightweight compute option for Pulsar ## Major changes diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/getting-started-standalone.md b/site2/website/versioned_docs/version-2.9.3-deprecated/getting-started-standalone.md index 90e3f820e74d1..573a33771d64e 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/getting-started-standalone.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/getting-started-standalone.md @@ -8,7 +8,7 @@ original_id: getting-started-standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > **Pulsar in production?** -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -65,7 +65,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -74,7 +74,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -122,7 +122,7 @@ pulsar-io-aerospike-@pulsar:version@.nar :::note * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -135,7 +135,7 @@ pulsar-io-aerospike-@pulsar:version@.nar ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -168,12 +168,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -205,7 +205,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/helm-install.md b/site2/website/versioned_docs/version-2.9.3-deprecated/helm-install.md index 80fb7f8d6c8d6..9f81f52e0dab1 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/helm-install.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/helm-install.md @@ -21,7 +21,7 @@ Before deploying Pulsar, you need to prepare your environment. ### Tools -Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools) on your computer. +Install [`helm`](helm-tools.md) and [`kubectl`](helm-tools.md) on your computer. ## Cloud cluster preparation @@ -31,8 +31,8 @@ To create and connect to the Kubernetes cluster, follow the instructions: ## Pulsar deployment -Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy). +Once the environment is set up and configuration is generated, you can now proceed to the [deployment of Pulsar](helm-deploy.md). ## Pulsar upgrade -To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade). +To upgrade an existing Kubernetes installation, follow the [upgrade documentation](helm-upgrade.md). diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/helm-overview.md b/site2/website/versioned_docs/version-2.9.3-deprecated/helm-overview.md index fc4cbfe426a93..125f595cbe68a 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/helm-overview.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/helm-overview.md @@ -52,9 +52,9 @@ Moreover, Helm chart supports: ## Quick start -To run with Apache Pulsar Helm chart as fast as possible in a **non-production** use case, we provide a [quick start guide](getting-started-helm) for Proof of Concept (PoC) deployments. +To run with Apache Pulsar Helm chart as fast as possible in a **non-production** use case, we provide a [quick start guide](getting-started-helm.md) for Proof of Concept (PoC) deployments. -This guide walks you through deploying Apache Pulsar Helm chart with default values and features, but it is *not* suitable for deployments in production-ready environments. To deploy the charts in production under sustained load, you can follow the complete [Installation Guide](helm-install). +This guide walks you through deploying Apache Pulsar Helm chart with default values and features, but it is *not* suitable for deployments in production-ready environments. To deploy the charts in production under sustained load, you can follow the complete [Installation Guide](helm-install.md). ## Troubleshooting @@ -64,7 +64,7 @@ Although we have done our best to make these charts as seamless as possible, tro The Apache Pulsar Helm chart contains all required dependencies. -If you deploy a PoC for testing, we strongly suggest you follow this [Quick Start Guide](getting-started-helm) for your first iteration. +If you deploy a PoC for testing, we strongly suggest you follow this [Quick Start Guide](getting-started-helm.md) for your first iteration. 1. [Preparation](helm-prepare.md) 2. [Deployment](helm-deploy.md) @@ -82,7 +82,7 @@ helm upgrade apache/pulsar -f pulsar.yaml ``` -For more detailed information, see [Upgrading](helm-upgrade). +For more detailed information, see [Upgrading](helm-upgrade.md). ## Uninstallation diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/helm-tools.md b/site2/website/versioned_docs/version-2.9.3-deprecated/helm-tools.md index efd61eae32506..6ba89006913b6 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/helm-tools.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/helm-tools.md @@ -25,7 +25,7 @@ You can get Helm from the project's [releases page](https://github.com/helm/helm ### Next steps -Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare). +Once kubectl and Helm are configured, you can configure your [Kubernetes cluster](helm-prepare.md). ## Additional information diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/helm-upgrade.md b/site2/website/versioned_docs/version-2.9.3-deprecated/helm-upgrade.md index e39cf97407355..7d671e6bfb3c1 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/helm-upgrade.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/helm-upgrade.md @@ -20,7 +20,7 @@ You can retrieve your previous `--set` arguments cleanly, with `helm get values To upgrade Apache Pulsar to a newer version, follow these steps: 1. Check the change log for the specific version you would like to upgrade to. -2. Go through [deployment documentation](helm-deploy) step by step. +2. Go through [deployment documentation](helm-deploy.md) step by step. 3. Extract your previous `--set` arguments with the following command. ```bash diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/io-cdc.md b/site2/website/versioned_docs/version-2.9.3-deprecated/io-cdc.md index 20f16ae0d6904..e6e662884826d 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/io-cdc.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/io-cdc.md @@ -13,8 +13,8 @@ Currently, Pulsar has the following CDC connectors. Name|Java Class |---|--- -[Canal source connector](io-canal-source)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) -[Debezium source connector](io-cdc-debezium)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • +[Canal source connector](io-canal-source.md)|[org.apache.pulsar.io.canal.CanalStringSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/java/org/apache/pulsar/io/canal/CanalStringSource.java) +[Debezium source connector](io-cdc-debezium.md)|
  • [org.apache.pulsar.io.debezium.DebeziumSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/core/src/main/java/org/apache/pulsar/io/debezium/DebeziumSource.java)
  • [org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/mysql/src/main/java/org/apache/pulsar/io/debezium/mysql/DebeziumMysqlSource.java)
  • [org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource.java](https://github.com/apache/pulsar/blob/master/pulsar-io/debezium/postgres/src/main/java/org/apache/pulsar/io/debezium/postgres/DebeziumPostgresSource.java)
    • For more information about Canal and Debezium, see the information below. diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/io-develop.md b/site2/website/versioned_docs/version-2.9.3-deprecated/io-develop.md index a20206fbf6a06..d6f4f8261ac82 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/io-develop.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/io-develop.md @@ -14,15 +14,15 @@ import TabItem from '@theme/TabItem'; This guide describes how to develop Pulsar connectors to move data between Pulsar and other systems. -Pulsar connectors are special [Pulsar Functions](functions-overview), so creating +Pulsar connectors are special [Pulsar Functions](functions-overview.md), so creating a Pulsar connector is similar to creating a Pulsar function. Pulsar connectors come in two types: | Type | Description | Example |---|---|--- -{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq) imports the messages of a RabbitMQ queue to a Pulsar topic. -{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis) exports the messages of a Pulsar topic to a Kinesis stream. +{@inject: github:Source:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Source.java}|Import data from another system to Pulsar.|[RabbitMQ source connector](io-rabbitmq.md) imports the messages of a RabbitMQ queue to a Pulsar topic. +{@inject: github:Sink:/pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java}|Export data from Pulsar to another system.|[Kinesis sink connector](io-kinesis.md) exports the messages of a Pulsar topic to a Kinesis stream. ## Develop @@ -275,7 +275,7 @@ For more information about **how to create integration tests for Pulsar connecto ## Package Once you've developed and tested your connector, you need to package it so that it can be submitted -to a [Pulsar Functions](functions-overview) cluster. +to a [Pulsar Functions](functions-overview.md) cluster. There are two methods to work with Pulsar Functions' runtime, that is, [NAR](#nar) and [uber JAR](#uber-jar). @@ -304,7 +304,7 @@ For more information about **how NAR works**, see [here](https://medium.com/hash ::: -Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors). +Pulsar uses the same mechanism for packaging **all** [built-in connectors](io-connectors.md). The easiest approach to package a Pulsar connector is to create a NAR package using [nifi-nar-maven-plugin](https://mvnrepository.com/artifact/org.apache.nifi/nifi-nar-maven-plugin). @@ -379,7 +379,7 @@ Pulsar connectors enable you to move data in and out of Pulsar easily. It is imp - Check the metrics provided by Pulsar. - Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring) guide. + Pulsar connectors expose the metrics that can be collected and used for monitoring the health of **Java** connectors. You can check the metrics by following the [monitoring](deploy-monitoring.md) guide. - Set and check your customized metrics. diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/io-overview.md b/site2/website/versioned_docs/version-2.9.3-deprecated/io-overview.md index 0070a015296db..3db5ee34042d3 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/io-overview.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/io-overview.md @@ -158,7 +158,7 @@ For more information about the options of `pulsar-admin sinks update`, see [here ## Work with connector -You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. +You can manage Pulsar connectors (for example, create, update, start, stop, restart, reload, delete and perform other operations on connectors) via the [Connector Admin CLI](reference-connector-admin.md) with [sources](io-cli.md#sources) and [sinks](io-cli.md#sinks) subcommands. -Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). +Connectors (sources and sinks) and Functions are components of instances, and they all run on Functions workers. When managing a source, sink or function via [Connector Admin CLI](reference-connector-admin.md) or [Functions Admin CLI](functions-cli.md), an instance is started on a worker. For more information, see [Functions worker](functions-worker.md#run-functions-worker-separately). diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/io-quickstart.md b/site2/website/versioned_docs/version-2.9.3-deprecated/io-quickstart.md index 774ab8bf911a1..b73063e75e0c3 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/io-quickstart.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/io-quickstart.md @@ -7,7 +7,7 @@ original_id: io-quickstart This tutorial provides a hands-on look at how you can move data out of Pulsar without writing a single line of code. -It is helpful to review the [concepts](io-overview) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. +It is helpful to review the [concepts](io-overview.md) for Pulsar I/O with running the steps in this guide to gain a deeper understanding. At the end of this tutorial, you are able to: @@ -17,7 +17,7 @@ At the end of this tutorial, you are able to: :::tip -* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone). However, all +* These instructions assume you are running Pulsar in [standalone mode](getting-started-standalone.md). However, all the commands used in this tutorial can be used in a multi-nodes Pulsar cluster without any changes. * All the instructions are assumed to run at the root directory of a Pulsar binary distribution. @@ -109,7 +109,7 @@ This section demonstrates how to connect Pulsar to Cassandra. :::tip * Make sure you have Docker installed. If you do not have one, see [install Docker](https://docs.docker.com/docker-for-mac/install/). -* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink). +* The Cassandra sink connector reads messages from Pulsar topics and writes the messages into Cassandra tables. For more information, see [Cassandra sink connector](io-cassandra-sink.md). ::: @@ -236,11 +236,11 @@ You can create a configuration file through one of the following methods. ``` -For more information, see [Cassandra sink connector](io-cassandra-sink). +For more information, see [Cassandra sink connector](io-cassandra-sink.md). ### Create a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on them. Run the following command to create a Cassandra sink connector with sink type _cassandra_ and the config file _examples/cassandra-sink.yml_ created previously. @@ -267,7 +267,7 @@ as a Pulsar Function and writes the messages produced in the topic _test_cassand ### Inspect a Cassandra sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * Get the information of a Cassandra sink. @@ -431,7 +431,7 @@ to monitor a connector and perform other operations on it. ### Delete a Cassandra Sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. ```bash @@ -455,7 +455,7 @@ This section demonstrates how to connect Pulsar to PostgreSQL. ::: and persists the messages to ClickHouse, MariaDB, PostgreSQL, or SQlite. ->For more information, see [JDBC sink connector](io-jdbc-sink). +>For more information, see [JDBC sink connector](io-jdbc-sink.md). ### Setup a PostgreSQL cluster @@ -615,7 +615,7 @@ In this section, you need to configure a JDBC sink connector. ### Create a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to create a sink connector and perform other operations on it. This example creates a sink connector and specifies the desired information. @@ -661,7 +661,7 @@ The sink has been created successfully if the following message appears. ### Inspect a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to monitor a connector and perform other operations on it. * List all running JDBC sink(s). @@ -781,7 +781,7 @@ to monitor a connector and perform other operations on it. ### Stop a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to stop a connector and perform other operations on it. ```bash @@ -809,7 +809,7 @@ The sink instance has been stopped successfully if the following message disappe ### Restart a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to restart a connector and perform other operations on it. ```bash @@ -845,7 +845,7 @@ Note that `pulsar-admin sinks localrun options` **runs a sink connector locally* ### Update a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to update a connector and perform other operations on it. This example updates the parallelism of the _pulsar-postgres-jdbc-sink_ sink connector to 2. @@ -913,7 +913,7 @@ The result shows that the parallelism is 2. ### Delete a JDBC sink -You can use the [Connector Admin CLI](io-cli) +You can use the [Connector Admin CLI](io-cli.md) to delete a connector and perform other operations on it. This example deletes the _pulsar-postgres-jdbc-sink_ sink connector. diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/io-use.md b/site2/website/versioned_docs/version-2.9.3-deprecated/io-use.md index b6409d4420ad7..da9ed746c4d37 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/io-use.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/io-use.md @@ -15,7 +15,7 @@ This guide describes how to use Pulsar connectors. ## Install a connector -Pulsar bundles several [builtin connectors](io-connectors) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. +Pulsar bundles several [builtin connectors](io-connectors.md) used to move data in and out of commonly used systems (such as database and messaging system). Optionally, you can create and use your desired non-builtin connectors. :::note diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/reference-cli-tools.md b/site2/website/versioned_docs/version-2.9.3-deprecated/reference-cli-tools.md index eff2625277504..6893da3ec6394 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/reference-cli-tools.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/reference-cli-tools.md @@ -7,7 +7,7 @@ original_id: reference-cli-tools Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more. -All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone). The following tools are currently documented: +All Pulsar command-line tools can be run from the `bin` directory of your [installed Pulsar package](getting-started-standalone.md). The following tools are currently documented: * [`pulsar`](#pulsar) * [`pulsar-client`](#pulsar-client) diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/reference-configuration.md b/site2/website/versioned_docs/version-2.9.3-deprecated/reference-configuration.md index c8705bbde828a..71f2eec9f97ee 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/reference-configuration.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/reference-configuration.md @@ -8,7 +8,7 @@ original_id: reference-configuration -You can manage Pulsar configuration by configuration files in the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a Pulsar [installation](getting-started-standalone). +You can manage Pulsar configuration by configuration files in the [`conf`](https://github.com/apache/pulsar/tree/master/conf) directory of a Pulsar [installation](getting-started-standalone.md). - [BookKeeper](#bookkeeper) - [Broker](#broker) diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/schema-get-started.md b/site2/website/versioned_docs/version-2.9.3-deprecated/schema-get-started.md index 97e270586a26d..afacb0fa51f2e 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/schema-get-started.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/schema-get-started.md @@ -17,7 +17,7 @@ Applications typically adopt one of the following approaches to guarantee type s #### Note > -> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp). +> Currently, the Pulsar schema registry is only available for the [Java client](client-libraries-java.md), [CGo client](client-libraries-cgo.md), [Python client](client-libraries-python.md), and [C++ client](client-libraries-cpp.md). ### Client-side approach diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/schema-manage.md b/site2/website/versioned_docs/version-2.9.3-deprecated/schema-manage.md index 4ae363882b2d5..c588aae619eee 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/schema-manage.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/schema-manage.md @@ -160,7 +160,7 @@ To manage schemas, you can use one of the following methods. | Method | Description | | --- | --- | -| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin). | +| **Admin CLI**
  • | You can use the `pulsar-admin` tool to manage Pulsar schemas, brokers, clusters, sources, sinks, topics, tenants and so on. For more information about how to use the `pulsar-admin` tool, see [here](reference-pulsar-admin.md). | | **REST API**
  • | Pulsar exposes schema related management API in Pulsar’s admin RESTful API. You can access the admin RESTful endpoint directly to manage schemas. For more information about how to use the Pulsar REST API, see [here](http://pulsar.apache.org/admin-rest-api/). | | **Java Admin API**
  • | Pulsar provides Java admin library. | diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/schema-understand.md b/site2/website/versioned_docs/version-2.9.3-deprecated/schema-understand.md index f454dd6873e8e..55bc662c66633 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/schema-understand.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/schema-understand.md @@ -479,7 +479,7 @@ Once a version is assigned/fetched to/for a schema, all subsequent messages prod The following example illustrates how the schema version works. -Suppose that a Pulsar [Java client](client-libraries-java) created using the code below attempts to connect to Pulsar and begins to send messages: +Suppose that a Pulsar [Java client](client-libraries-java.md) created using the code below attempts to connect to Pulsar and begins to send messages: ```java diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/security-athenz.md b/site2/website/versioned_docs/version-2.9.3-deprecated/security-athenz.md index 947c3f470be46..8a39fe25316d0 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/security-athenz.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/security-athenz.md @@ -76,7 +76,7 @@ For more information on Pulsar client authentication using Athenz, see the follo ## Configure CLI tools for Athenz -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following authentication parameters to the `conf/client.conf` config file to use Athenz with CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/security-authorization.md b/site2/website/versioned_docs/version-2.9.3-deprecated/security-authorization.md index cc9c03d5fc1dd..9cfd7c8c203f6 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/security-authorization.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/security-authorization.md @@ -27,7 +27,7 @@ superUserRoles=my-super-user-1,my-super-user-2 > A full list of parameters is available in the `conf/broker.conf` file. > You can also find the default values for those parameters in [Broker Configuration](reference-configuration.md#broker). -Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication), every broker needs to be able to publish to all the other topics of clusters. +Typically, you use superuser roles for administrators, clients as well as broker-to-broker authorization. When you use [geo-replication](concepts-replication.md), every broker needs to be able to publish to all the other topics of clusters. You can also enable the authorization for the proxy in the proxy configuration file (`conf/proxy.conf`). Once you enable the authorization on the proxy, the proxy does an additional authorization check before forwarding the request to a broker. If you enable authorization on the broker, the broker checks the authorization of the request when the broker receives the forwarded request. @@ -59,7 +59,7 @@ superUserRoles=my-super-user-1,my-super-user-2,my-proxy-role Pulsar [instance](reference-terminology.md#instance) administrators or some kind of self-service portal typically provisions a Pulsar [tenant](reference-terminology.md#tenant). -You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin) tool. +You can manage tenants using the [`pulsar-admin`](reference-pulsar-admin.md) tool. ### Create a new tenant @@ -87,7 +87,7 @@ persistent://tenant/namespace/topic ### Manage permissions -You can use [Pulsar Admin Tools](admin-api-permissions) for managing permission in Pulsar. +You can use [Pulsar Admin Tools](admin-api-permissions.md) for managing permission in Pulsar. ### Pulsar admin authentication @@ -117,7 +117,7 @@ PulsarAdmin admin = PulsarAdmin.builder() When a client is identified with multiple roles in a token (the type of role claim in the token is an array) during the authentication process, Pulsar supports to check the permissions of all the roles and further authorize the client as long as one of its roles has the required permissions. > **Note**
    -> This authorization method is only compatible with [JWT authentication](security-jwt). +> This authorization method is only compatible with [JWT authentication](security-jwt.md). To enable this authorization method, configure the authorization provider as `MultiRolesTokenAuthorizationProvider` in the `conf/broker.conf` file. diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/security-bouncy-castle.md b/site2/website/versioned_docs/version-2.9.3-deprecated/security-bouncy-castle.md index abc25c71b8ae2..be937055d8e31 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/security-bouncy-castle.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/security-bouncy-castle.md @@ -16,7 +16,7 @@ In Pulsar, security and crypto have dependencies on BouncyCastle Jars. For the d `Bouncy Castle` provides both [FIPS](https://www.bouncycastle.org/fips_faq.html) and non-FIPS version. But in a JVM, you can not include both of the 2 versions, and you need to exclude the current version before include the other. -In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. +In Pulsar, the security and crypto methods also depends on `Bouncy Castle`, especially in [TLS Authentication](security-tls-authentication.md) and [Transport Encryption](security-encryption.md). This document contains the configuration between BouncyCastle FIPS(BC-FIPS) and non-FIPS(BC-non-FIPS) version while using Pulsar. ## How BouncyCastle modules packaged in Pulsar diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/security-extending.md b/site2/website/versioned_docs/version-2.9.3-deprecated/security-extending.md index c088e3a2c1c2c..e7484453b8beb 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/security-extending.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/security-extending.md @@ -9,7 +9,7 @@ Pulsar provides a way to use custom authentication and authorization mechanisms. ## Authentication -Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview). +Pulsar supports mutual TLS and Athenz authentication plugins. For how to use these authentication plugins, you can refer to the description in [Security](security-overview.md). You can use a custom authentication mechanism by providing the implementation in the form of two plugins. One plugin is for the Client library and the other plugin is for the Pulsar Proxy and/or Pulsar Broker to validate the credentials. diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/security-jwt.md b/site2/website/versioned_docs/version-2.9.3-deprecated/security-jwt.md index b69c7bcb7b20a..fcee5c0ce21da 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/security-jwt.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/security-jwt.md @@ -32,11 +32,11 @@ Application specifies the token when you create the client instance. An alternat > #### Always use TLS transport encryption > Sending a token is equivalent to sending a password over the wire. You had better use TLS encryption all the time when you connect to the Pulsar service. See -> [Transport Encryption using TLS](security-tls-transport) for more details. +> [Transport Encryption using TLS](security-tls-transport.md) for more details. ### CLI Tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use the token authentication with CLI tools of Pulsar: @@ -303,7 +303,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, add the following parameters to `proxy.conf`: -The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization). +The proxy uses its own token when connecting to brokers. You need to configure the role token for this key pair in the `proxyRoles` of the brokers. For more details, see the [authorization guide](security-authorization.md). ```properties diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/security-kerberos.md b/site2/website/versioned_docs/version-2.9.3-deprecated/security-kerberos.md index 670586a9d1691..c49fa3bea1fce 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/security-kerberos.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/security-kerberos.md @@ -373,7 +373,7 @@ saslJaasBrokerSectionName=PulsarBroker ## Regarding authorization and role token -For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization). +For Kerberos authentication, we usually use the authenticated principal as the role token for Pulsar authorization. For more information of authorization in Pulsar, see [security authorization](security-authorization.md). If you enable 'authorizationEnabled', you need to set `superUserRoles` in `broker.conf` that corresponds to the name registered in kdc. diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/security-overview.md b/site2/website/versioned_docs/version-2.9.3-deprecated/security-overview.md index 23d4477214cb7..df630ab519bb4 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/security-overview.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/security-overview.md @@ -21,7 +21,7 @@ You had better secure the service components in your Apache Pulsar deployment. In Pulsar, a *role* is a string, like `admin` or `app1`, which can represent a single client or multiple clients. You can use roles to control permission for clients to produce or consume from certain topics, administer the configuration for tenants, and so on. -Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization) to determine what the client is authorized to do. +Apache Pulsar uses a [Authentication Provider](#authentication-providers) to establish the identity of a client and then assign a *role token* to that client. This role token is then used for [Authorization and ACLs](security-authorization.md) to determine what the client is authorized to do. ## Authentication providers diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/security-tls-authentication.md b/site2/website/versioned_docs/version-2.9.3-deprecated/security-tls-authentication.md index f3c32a05d9751..85d2240f41306 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/security-tls-authentication.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/security-tls-authentication.md @@ -7,9 +7,9 @@ original_id: security-tls-authentication ## TLS authentication overview -TLS authentication is an extension of [TLS transport encryption](security-tls-transport). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. +TLS authentication is an extension of [TLS transport encryption](security-tls-transport.md). Not only servers have keys and certs that the client uses to verify the identity of servers, clients also have keys and certs that the server uses to verify the identity of clients. You must have TLS transport encryption configured on your cluster before you can use TLS authentication. This guide assumes you already have TLS transport encryption configured. -`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides TLS related cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ### Create client certificates @@ -103,7 +103,7 @@ brokerClientTrustCertsFilePath=/path/my-ca/certs/ca.cert.pem To configure proxies to authenticate clients, add the following parameters to `proxy.conf`, alongside [the configuration to enable tls transport](security-tls-transport.md#proxy-configuration): -The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +The proxy should have its own client key pair for connecting to brokers. You need to configure the role token for this key pair in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties @@ -123,7 +123,7 @@ When you use TLS authentication, client connects via TLS transport. You need to ### CLI tools -[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-pulsar-admin.md), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS authentication with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/security-tls-keystore.md b/site2/website/versioned_docs/version-2.9.3-deprecated/security-tls-keystore.md index 6cc3a2c60835e..0b3b50fcebb10 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/security-tls-keystore.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/security-tls-keystore.md @@ -7,7 +7,7 @@ original_id: security-tls-keystore ## Overview -Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication) between clients and Apache Pulsar service. +Apache Pulsar supports [TLS encryption](security-tls-transport.md) and [TLS authentication](security-tls-authentication.md) between clients and Apache Pulsar service. By default it uses PEM format file configuration. This page tries to describe use [KeyStore](https://en.wikipedia.org/wiki/Java_KeyStore) type configure for TLS. @@ -185,7 +185,7 @@ This is similar to [TLS encryption configuing for client with PEM type](security For a a minimal configuration, user need to provide the TrustStore information. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties @@ -278,7 +278,7 @@ webSocketServiceEnabled=false Besides the TLS encryption configuring. The main work is configuring the KeyStore, which contains a valid CN as client role, for client. e.g. -1. for [Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +1. for [Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools#pulsar-admin), [`pulsar-perf`](reference-cli-tools#pulsar-perf), and [`pulsar-client`](reference-cli-tools#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. ```properties diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/security-tls-transport.md b/site2/website/versioned_docs/version-2.9.3-deprecated/security-tls-transport.md index 92e8dc3c3d280..2cad17a78c350 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/security-tls-transport.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/security-tls-transport.md @@ -9,7 +9,7 @@ original_id: security-tls-transport By default, Apache Pulsar clients communicate with the Apache Pulsar service in plain text. This means that all data is sent in the clear. You can use TLS to encrypt this traffic to protect the traffic from the snooping of a man-in-the-middle attacker. -You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz) on top of TLS transport encryption. +You can also configure TLS for both encryption and authentication. Use this guide to configure just TLS transport encryption and refer to [here](security-tls-authentication.md) for TLS authentication configuration. Alternatively, you can use [another authentication mechanism](security-athenz.md) on top of TLS transport encryption. > Note that enabling TLS may impact the performance due to encryption overhead. @@ -19,7 +19,7 @@ TLS is a form of [public key cryptography](https://en.wikipedia.org/wiki/Public- To use TLS transport encryption, you need two kinds of key pairs, **server key pairs** and a **certificate authority**. -You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication). +You can use a third kind of key pair, **client key pairs**, for [client authentication](security-tls-authentication.md). You should store the **certificate authority** private key in a very secure location (a fully encrypted, disconnected, air gapped computer). As for the certificate authority public key, the **trust cert**, you can freely shared it. @@ -27,9 +27,9 @@ For both client and server key pairs, the administrator first generates a privat For TLS transport encryption, the clients can use the **trust cert** to verify that the server has a key pair that the certificate authority signed when the clients are talking to the server. A man-in-the-middle attacker does not have access to the certificate authority, so they couldn't create a server with such a key pair. -For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview)). +For TLS authentication, the server uses the **trust cert** to verify that the client has a key pair that the certificate authority signed. The common name of the **client cert** is then used as the client's role token (see [Overview](security-overview.md)). -`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle). +`Bouncy Castle Provider` provides cipher suites and algorithms in Pulsar. If you need [FIPS](https://www.bouncycastle.org/fips_faq.html) version of `Bouncy Castle Provider`, please reference [Bouncy Castle page](security-bouncy-castle.md). ## Create TLS certificates @@ -130,7 +130,7 @@ At this point, you have a cert, `broker.cert.pem`, and a key, `broker.key-pk8.pe ## Configure broker -To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone). +To configure a Pulsar [broker](reference-terminology.md#broker) to use TLS transport encryption, you need to make some changes to `broker.conf`, which locates in the `conf` directory of your [Pulsar installation](getting-started-standalone.md). Add these values to the configuration file (substituting the appropriate certificate paths where necessary): @@ -202,7 +202,7 @@ The examples below show that hostname verification is disabled for the CLI tools ### CLI tools -[Command-line tools](reference-cli-tools) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. +[Command-line tools](reference-cli-tools.md) like [`pulsar-admin`](reference-cli-tools.md#pulsar-admin), [`pulsar-perf`](reference-cli-tools.md#pulsar-perf), and [`pulsar-client`](reference-cli-tools.md#pulsar-client) use the `conf/client.conf` config file in a Pulsar installation. You need to add the following parameters to that file to use TLS transport with the CLI tools of Pulsar: diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/security-token-admin.md b/site2/website/versioned_docs/version-2.9.3-deprecated/security-token-admin.md index 1a438dc7923ee..a265f6320d28f 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/security-token-admin.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/security-token-admin.md @@ -163,7 +163,7 @@ tokenSecretKey=file:///path/to/secret.key To configure proxies to authenticate clients, put the following in `proxy.conf`: The proxy will have its own token used when talking to brokers. The role token for this -key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization) for more details. +key pair should be configured in the ``proxyRoles`` of the brokers. See the [authorization guide](security-authorization.md) for more details. ```properties diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/sql-deployment-configurations.md b/site2/website/versioned_docs/version-2.9.3-deprecated/sql-deployment-configurations.md index 67f7ab17c77d0..02d8bc78f6cb9 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/sql-deployment-configurations.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/sql-deployment-configurations.md @@ -64,7 +64,7 @@ Since Pulsar SQL is powered by [Trino (formerly Presto SQL)](https://trino.io), :::note -For how to set up a standalone single node environment, refer to [Query data](sql-getting-started). +For how to set up a standalone single node environment, refer to [Query data](sql-getting-started.md). ::: diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/sql-overview.md b/site2/website/versioned_docs/version-2.9.3-deprecated/sql-overview.md index 4a4d5f01b0bc5..0235d0256c5f1 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/sql-overview.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/sql-overview.md @@ -5,7 +5,7 @@ sidebar_label: "Overview" original_id: sql-overview --- -Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL)](https://trino.io/). +Apache Pulsar is used to store streams of event data, and the event data is structured with predefined fields. With the implementation of the [Schema Registry](schema-get-started.md), you can store structured data in Pulsar and query the data by using [Trino (formerly Presto SQL.md)](https://trino.io/). As the core of Pulsar SQL, Presto Pulsar connector enables Presto workers within a Presto cluster to query data from Pulsar. diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/standalone.md b/site2/website/versioned_docs/version-2.9.3-deprecated/standalone.md index 798ba379d2d18..a487be92adfcf 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/standalone.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/standalone.md @@ -8,7 +8,7 @@ original_id: standalone For local development and testing, you can run Pulsar in standalone mode on your machine. The standalone mode includes a Pulsar broker, the necessary ZooKeeper and BookKeeper components running inside of a single Java Virtual Machine (JVM) process. > **Pulsar in production?** -> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal) guide. +> If you're looking to run a full production Pulsar installation, see the [Deploying a Pulsar instance](deploy-bare-metal.md) guide. ## Install Pulsar standalone @@ -65,7 +65,7 @@ Directory | Contains :---------|:-------- `bin` | Pulsar's command-line tools, such as [`pulsar`](reference-cli-tools.md#pulsar) and [`pulsar-admin`](https://pulsar.apache.org/tools/pulsar-admin/). `conf` | Configuration files for Pulsar, including [broker configuration](reference-configuration.md#broker), [ZooKeeper configuration](reference-configuration.md#zookeeper), and more. -`examples` | A Java JAR file containing [Pulsar Functions](functions-overview) example. +`examples` | A Java JAR file containing [Pulsar Functions](functions-overview.md) example. `lib` | The [JAR](https://en.wikipedia.org/wiki/JAR_(file_format)) files used by Pulsar. `licenses` | License files, in the`.txt` form, for various components of the Pulsar [codebase](https://github.com/apache/pulsar). @@ -74,7 +74,7 @@ These directories are created once you begin running Pulsar. Directory | Contains :---------|:-------- `data` | The data storage directory used by ZooKeeper and BookKeeper. -`instances` | Artifacts created for [Pulsar Functions](functions-overview). +`instances` | Artifacts created for [Pulsar Functions](functions-overview.md). `logs` | Logs created by the installation. :::tip @@ -122,7 +122,7 @@ pulsar-io-aerospike-@pulsar:version@.nar :::note * If you are running Pulsar in a bare metal cluster, make sure `connectors` tarball is unzipped in every pulsar directory of the broker (or in every pulsar directory of function-worker if you are running a separate worker cluster for Pulsar Functions). -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled [all builtin connectors](io-overview.md#working-with-connectors). ::: @@ -135,7 +135,7 @@ pulsar-io-aerospike-@pulsar:version@.nar ::: -To get started with [tiered storage offloaders](concepts-tiered-storage), you need to download the offloaders tarball release on every broker node in one of the following ways: +To get started with [tiered storage offloaders](concepts-tiered-storage.md), you need to download the offloaders tarball release on every broker node in one of the following ways: * download from the Apache mirror Pulsar Tiered Storage Offloaders @pulsar:version@ release @@ -168,12 +168,12 @@ tiered-storage-jcloud-@pulsar:version@.nar ``` -For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage). +For more information on how to configure tiered storage, see [Tiered storage cookbook](cookbooks-tiered-storage.md). :::note * If you are running Pulsar in a bare metal cluster, make sure that `offloaders` tarball is unzipped in every broker's pulsar directory. -* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. +* If you are [running Pulsar in Docker](getting-started-docker.md) or deploying Pulsar using a docker image (e.g. [K8S](deploy-kubernetes.md) or [DC/OS](https://dcos.io/)), you can use the `apachepulsar/pulsar-all` image instead of the `apachepulsar/pulsar` image. `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. ::: @@ -205,7 +205,7 @@ If you have started Pulsar successfully, you will see `INFO`-level log messages You can also run the service as a background process using the `pulsar-daemon start standalone` command. For more information, see [pulsar-daemon](https://pulsar.apache.org/docs/en/reference-cli-tools/#pulsar-daemon). > -> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview) document to secure your deployment. +> * By default, there is no encryption, authentication, or authorization configured. Apache Pulsar can be accessed from remote server without any authorization. Please do check [Security Overview](security-overview.md) document to secure your deployment. > > * When you start a local standalone cluster, a `public/default` [namespace](concepts-messaging.md#namespaces) is created automatically. The namespace is used for development purposes. All Pulsar topics are managed within namespaces. For more information, see [Topics](concepts-messaging.md#topics). diff --git a/site2/website/versioned_docs/version-2.9.3-deprecated/tiered-storage-overview.md b/site2/website/versioned_docs/version-2.9.3-deprecated/tiered-storage-overview.md index 7c6528ce2b36c..c635034f463b4 100644 --- a/site2/website/versioned_docs/version-2.9.3-deprecated/tiered-storage-overview.md +++ b/site2/website/versioned_docs/version-2.9.3-deprecated/tiered-storage-overview.md @@ -13,9 +13,9 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws). + - For more information about how to use the AWS S3 offloader with Pulsar, see [here](tiered-storage-aws.md). - - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs). + - For more information about how to use the GCS offloader with Pulsar, see [here](tiered-storage-gcs.md). ::: @@ -25,7 +25,7 @@ Pulsar's **Tiered Storage** feature allows older backlog data to be moved from B :::tip - For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem). + For more information about how to use the filesystem offloader with Pulsar, see [here](tiered-storage-filesystem.md). :::