From 9c4d44c256f0be0e23c205d548ce9976923a822e Mon Sep 17 00:00:00 2001 From: liangyepianzhou Date: Tue, 27 Sep 2022 19:41:18 +0800 Subject: [PATCH 1/5] [fxi][doc] Rewrite the steps of docker-deploy ### Motivation The old implementation is incompletely. ### Modification Rewrite it. --- site2/docs/deploy-docker.md | 129 ++++++++++++++++++++++++++++-------- 1 file changed, 103 insertions(+), 26 deletions(-) diff --git a/site2/docs/deploy-docker.md b/site2/docs/deploy-docker.md index b8733d14baccc..5efc0415316e2 100644 --- a/site2/docs/deploy-docker.md +++ b/site2/docs/deploy-docker.md @@ -3,50 +3,127 @@ id: deploy-docker title: Deploy a cluster on Docker sidebar_label: "Docker" --- - -To deploy a Pulsar cluster on Docker, complete the following steps: -1. Deploy a ZooKeeper cluster (optional) -2. Initialize cluster metadata -3. Deploy a BookKeeper cluster -4. Deploy one or more Pulsar brokers +## Deploy a cluster on Docker +To deploy a Pulsar cluster on Docker, you need to complete the next steps: +1. pull the pulsar docker image +2. create the zookeeper, bookie, broker container by the image +3. modify the broker.conf and bookkeeper.conf +4. create a network, and make the container connects to it. +5. start the zookeeper, and then init the cluster metadata +6. start the broker and bookie ## Prepare +To run Pulsar on Docker, you need to create a container for each Pulsar component: ZooKeeper, BookKeeper and broker. You can pull the images of ZooKeeper and BookKeeper separately on Docker Hub, and pull a Pulsar image for the broker. You can also pull only one Pulsar image and create three containers with this image. This tutorial takes the second option as an example. -To run Pulsar on Docker, you need to create a container for each Pulsar component: ZooKeeper, BookKeeper and broker. You can pull the images of ZooKeeper and BookKeeper separately on [Docker Hub](https://hub.docker.com/), and pull a [Pulsar image](https://hub.docker.com/r/apachepulsar/pulsar-all/tags) for the broker. You can also pull only one [Pulsar image](https://hub.docker.com/r/apachepulsar/pulsar-all/tags) and create three containers with this image. This tutorial takes the second option as an example. - -### Pull a Pulsar image -You can pull a Pulsar image from [Docker Hub](https://hub.docker.com/r/apachepulsar/pulsar-all/tags) with the following command. +## Pull a Pulsar image +You can pull a Pulsar image from Docker Hub with the following command. If you want to use some connectors, you can use apachepulsar/pulsar-all there. -```shell +```java docker pull apachepulsar/pulsar-all:latest ``` +## Create containers +* Create zookeeper container + +``` +docker run -it --privileged=true -u=root --name zookeeper apachepulsar/pulsar-all:latest /bin/bash +``` -### Create three containers -Create containers for ZooKeeper, BookKeeper and broker. In this example, they are named as `zookeeper`, `bookkeeper` and `broker` respectively. You can name them as you want with the `--name` flag. By default, the container names are created randomly. +* Create broker container -```shell -docker run -it --name bookkeeper apachepulsar/pulsar-all:latest /bin/bash -docker run -it --name zookeeper apachepulsar/pulsar-all:latest /bin/bash -docker run -it --name broker apachepulsar/pulsar-all:latest /bin/bash +``` +docker run -it --privileged=true -u=root --name broker apachepulsar/pulsar-all:latest /bin/bash ``` -### Create a network -To deploy a Pulsar cluster on Docker, you need to create a `network` and connect the containers of ZooKeeper, BookKeeper and broker to this network. The following command creates the network `pulsar`: +* Create bookie container -```shell -docker network create pulsar ``` +docker run -it --privileged=true -u=root --name bookie apachepulsar/pulsar-all:latest /bin/bash +``` +## Modify the configurations + +1. Copy the configuration of broker from docker container to local. -### Connect containers to network -Connect the containers of ZooKeeper, BookKeeper and broker to the `pulsar` network with the following commands. +``` +sudo docker cp broker:/pulsar/conf/broker.conf ./broker.conf +``` + +2. Modify the `broker.conf` + * metadataurl = zookeeper:2181 + * cluster-name = cluster-a + * managedLedgerDefaultEnsembleSize=1 + * managedLedgerDefaultWriteQuorum=1 + * managedLedgerDefaultAckQuorum=1 +3. Move the broker.conf to zookeeper and broker container +``` +sudo docker cp ./broker.conf zookeeper:/pulsar/conf/ +``` + +``` +sudo docker cp ./broker.conf broker:/pulsar/conf/ +``` +4. Modify the `bookkeeper.conf` + * zkServers=zookeeper:2181 + * metadataServiceUri=metadata-store:zk:zookeeper:2181 +5. Move the bookkeeper.conf to the bookie container +``` +sudo docker cp ./bookkeeper.conf bookie:/pulsar/conf/ +``` +## Create the network +To deploy a Pulsar cluster on Docker, you need to create a network and connect the containers of ZooKeeper, BookKeeper and broker to this network. The following command creates the network pulsar: +``` +docker network create pulsar +``` +Connect the containers of ZooKeeper, BookKeeper and broker to the pulsar network with the following commands. -```shell +``` docker network connect pulsar zookeeper +``` +``` docker network connect pulsar bookkeeper +``` +``` docker network connect pulsar broker +``` +## start the zookeeper, and init the cluster metadata +Start the zookeeper service in the docker container by the following commands: +1. Open the zookeeper container +``` +docker exec -it zookeeper bash +``` +2. Start the zookeeper service +``` +bin/pulsar-daemon start zookeeper +``` +3. init the cluster metadata +``` + bin/pulsar initialize-cluster-metadata \ + --cluster cluster-a \ + --zookeeper zookeeper:2181 \ + --configuration-store zookeeper:2181 \ + --web-service-url http://broker:8080 \ + --broker-service-url pulsar://broker:6650 \ +``` +## start the broker and bookie service +Start the broker and bookie service in the docker container by the following commands: +### Start broker service +1. Open the broker container +``` +docker exec -it broker bash +``` +2. Start the broker service +``` +bin/pulsar-daemon start broker +``` + +### Start bookie service +1. Open the bookie container +``` +docker exec -it bookie bash +``` +2. Start the bookie service +``` +bin/pulsar-daemon start bookie ``` -To check whether the containers are successfully connected to the network, enter the `docker network inspect pulsar` command. -For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal.md). From b5384389b1b86437022286f47f2d3aa8777b7407 Mon Sep 17 00:00:00 2001 From: liangyepianzhou Date: Thu, 29 Sep 2022 16:54:22 +0800 Subject: [PATCH 2/5] replace conf with env --- site2/docs/deploy-docker.md | 42 ++++++------------------------------- 1 file changed, 6 insertions(+), 36 deletions(-) diff --git a/site2/docs/deploy-docker.md b/site2/docs/deploy-docker.md index 5efc0415316e2..47eaf8a3e3070 100644 --- a/site2/docs/deploy-docker.md +++ b/site2/docs/deploy-docker.md @@ -7,10 +7,9 @@ sidebar_label: "Docker" To deploy a Pulsar cluster on Docker, you need to complete the next steps: 1. pull the pulsar docker image 2. create the zookeeper, bookie, broker container by the image -3. modify the broker.conf and bookkeeper.conf -4. create a network, and make the container connects to it. -5. start the zookeeper, and then init the cluster metadata -6. start the broker and bookie +3. create a network, and make the container connects to it. +4. start the zookeeper, and then init the cluster metadata +5. start the broker and bookie ## Prepare To run Pulsar on Docker, you need to create a container for each Pulsar component: ZooKeeper, BookKeeper and broker. You can pull the images of ZooKeeper and BookKeeper separately on Docker Hub, and pull a Pulsar image for the broker. You can also pull only one Pulsar image and create three containers with this image. This tutorial takes the second option as an example. @@ -25,48 +24,19 @@ docker pull apachepulsar/pulsar-all:latest * Create zookeeper container ``` -docker run -it --privileged=true -u=root --name zookeeper apachepulsar/pulsar-all:latest /bin/bash +docker run -d --privileged=true -u=root -e PULSAR_PREFIX_metadataStoreUrl=zk:zookeeper:2181 -e PULSAR_PREFIX_cluster-name=cluster-a -e PULSAR_PREFIX_managedLedgerDefaultEnsembleSize=1 -e PULSAR_PREFIX_managedLedgerDefaultWriteQuorum=1 -e PULSAR_PREFIX_managedLedgerDefaultAckQuorum=1 --name zookeeper apachepulsar/pulsar-all:latest /bin/bash ``` * Create broker container ``` -docker run -it --privileged=true -u=root --name broker apachepulsar/pulsar-all:latest /bin/bash +docker run -d --privileged=true -u=root -e PULSAR_PREFIX_metadataStoreUrl=zk:zookeeper:2181 -e PULSAR_PREFIX_cluster-name=cluster-a -e PULSAR_PREFIX_managedLedgerDefaultEnsembleSize=1 -e PULSAR_PREFIX_managedLedgerDefaultWriteQuorum=1 -e PULSAR_PREFIX_managedLedgerDefaultAckQuorum=1 --name broker apachepulsar/pulsar-all:latest /bin/bash ``` * Create bookie container ``` -docker run -it --privileged=true -u=root --name bookie apachepulsar/pulsar-all:latest /bin/bash -``` -## Modify the configurations - -1. Copy the configuration of broker from docker container to local. - -``` -sudo docker cp broker:/pulsar/conf/broker.conf ./broker.conf -``` - -2. Modify the `broker.conf` - * metadataurl = zookeeper:2181 - * cluster-name = cluster-a - * managedLedgerDefaultEnsembleSize=1 - * managedLedgerDefaultWriteQuorum=1 - * managedLedgerDefaultAckQuorum=1 -3. Move the broker.conf to zookeeper and broker container -``` -sudo docker cp ./broker.conf zookeeper:/pulsar/conf/ -``` - -``` -sudo docker cp ./broker.conf broker:/pulsar/conf/ -``` -4. Modify the `bookkeeper.conf` - * zkServers=zookeeper:2181 - * metadataServiceUri=metadata-store:zk:zookeeper:2181 -5. Move the bookkeeper.conf to the bookie container -``` -sudo docker cp ./bookkeeper.conf bookie:/pulsar/conf/ +docker run -d --privileged=true -u=root -e PULSAR_PREFIX_zkServers=zookeeper:2181 -e PULSAR_PREFIX_metadataServiceUri=metadata-store:zk:zookeeper:2181 --name bookie apachepulsar/pulsar-all:latest /bin/bash ``` ## Create the network To deploy a Pulsar cluster on Docker, you need to create a network and connect the containers of ZooKeeper, BookKeeper and broker to this network. The following command creates the network pulsar: From 12e751e471f6c72df9509a9363919806c254027f Mon Sep 17 00:00:00 2001 From: liangyepianzhou Date: Mon, 3 Oct 2022 03:01:59 +0800 Subject: [PATCH 3/5] optimize the doc and add a docker-compose.yaml to deploy Pulsar cluster --- site2/docs/deploy-docker.md | 206 +++++++++++++++++++++++++++++------- 1 file changed, 167 insertions(+), 39 deletions(-) diff --git a/site2/docs/deploy-docker.md b/site2/docs/deploy-docker.md index 47eaf8a3e3070..ad3184d7c61cd 100644 --- a/site2/docs/deploy-docker.md +++ b/site2/docs/deploy-docker.md @@ -4,12 +4,13 @@ title: Deploy a cluster on Docker sidebar_label: "Docker" --- ## Deploy a cluster on Docker -To deploy a Pulsar cluster on Docker, you need to complete the next steps: -1. pull the pulsar docker image -2. create the zookeeper, bookie, broker container by the image -3. create a network, and make the container connects to it. -4. start the zookeeper, and then init the cluster metadata -5. start the broker and bookie +To deploy a Pulsar cluster on Docker, you need to complete the following steps: +1. Pull the Pulsar Docker image. +2. Create the zookeeper, bookie, broker container by the image +3. Modify the broker.conf and bookkeeper.conf +4. Create ZooKeeper, bookie, and broker containers. +5. Start ZooKeeper and initialize cluster metadata. +6. Start brokers and bookies. ## Prepare To run Pulsar on Docker, you need to create a container for each Pulsar component: ZooKeeper, BookKeeper and broker. You can pull the images of ZooKeeper and BookKeeper separately on Docker Hub, and pull a Pulsar image for the broker. You can also pull only one Pulsar image and create three containers with this image. This tutorial takes the second option as an example. @@ -21,79 +22,206 @@ You can pull a Pulsar image from Docker Hub with the following command. If you w docker pull apachepulsar/pulsar-all:latest ``` ## Create containers -* Create zookeeper container - -``` -docker run -d --privileged=true -u=root -e PULSAR_PREFIX_metadataStoreUrl=zk:zookeeper:2181 -e PULSAR_PREFIX_cluster-name=cluster-a -e PULSAR_PREFIX_managedLedgerDefaultEnsembleSize=1 -e PULSAR_PREFIX_managedLedgerDefaultWriteQuorum=1 -e PULSAR_PREFIX_managedLedgerDefaultAckQuorum=1 --name zookeeper apachepulsar/pulsar-all:latest /bin/bash +Create a ZooKeeper container +```bash +docker run -d --privileged=true -u=root -p 2181:2181 -e metadataStoreUrl=zk:zookeeper:2181 -e cluster-name=cluster-a -e managedLedgerDefaultEnsembleSize=1 -e managedLedgerDefaultWriteQuorum=1 -e managedLedgerDefaultAckQuorum=1 --name zookeeper apachepulsar/pulsar-all:latest /bin/bash ``` * Create broker container -``` -docker run -d --privileged=true -u=root -e PULSAR_PREFIX_metadataStoreUrl=zk:zookeeper:2181 -e PULSAR_PREFIX_cluster-name=cluster-a -e PULSAR_PREFIX_managedLedgerDefaultEnsembleSize=1 -e PULSAR_PREFIX_managedLedgerDefaultWriteQuorum=1 -e PULSAR_PREFIX_managedLedgerDefaultAckQuorum=1 --name broker apachepulsar/pulsar-all:latest /bin/bash +```bash +docker run -d --privileged=true -u=root -p 6650:6650 -p 8080:8080 -e metadataStoreUrl=zk:zookeeper:2181 -e zookeeperServers=zookeeper:2181 -e clusterName=cluster-a -e managedLedgerDefaultEnsembleSize=1 -e managedLedgerDefaultWriteQuorum=1 -e managedLedgerDefaultAckQuorum=1 --name broker apachepulsar/pulsar-all:latest /bin/bash ``` -* Create bookie container +* Create a bookie container -``` -docker run -d --privileged=true -u=root -e PULSAR_PREFIX_zkServers=zookeeper:2181 -e PULSAR_PREFIX_metadataServiceUri=metadata-store:zk:zookeeper:2181 --name bookie apachepulsar/pulsar-all:latest /bin/bash +```bash +docker run -d --privileged=true -u=root -e clusterName=cluster-a -e zkServers=zookeeper:2181 -e metadataServiceUri=metadata-store:zk:zookeeper:2181 --name bookie apachepulsar/pulsar-all:latest /bin/bash ``` ## Create the network To deploy a Pulsar cluster on Docker, you need to create a network and connect the containers of ZooKeeper, BookKeeper and broker to this network. The following command creates the network pulsar: -``` +```bash docker network create pulsar ``` Connect the containers of ZooKeeper, BookKeeper and broker to the pulsar network with the following commands. -``` +```bash docker network connect pulsar zookeeper ``` -``` +```bash docker network connect pulsar bookkeeper ``` -``` +```bash docker network connect pulsar broker ``` -## start the zookeeper, and init the cluster metadata -Start the zookeeper service in the docker container by the following commands: -1. Open the zookeeper container -``` +## Start ZooKeeper and initialize cluster metadata +Start the ZooKeeper service in the Docker container by running the following commands. +1. Open the ZooKeeper container. +```bash docker exec -it zookeeper bash ``` -2. Start the zookeeper service -``` -bin/pulsar-daemon start zookeeper + +2. copy configuration from env +```bash +bin/apply-config-from-env.py conf/zookeeper.conf ``` -3. init the cluster metadata + +```bash +bin/generate-zookeeper-config.sh conf/zookeeper.conf ``` - bin/pulsar initialize-cluster-metadata \ - --cluster cluster-a \ - --zookeeper zookeeper:2181 \ - --configuration-store zookeeper:2181 \ - --web-service-url http://broker:8080 \ - --broker-service-url pulsar://broker:6650 \ + +3. Start the ZooKeeper service. +```bash +bin/pulsar-daemon start zookeeper ``` -## start the broker and bookie service -Start the broker and bookie service in the docker container by the following commands: + +4. Initialize the cluster metadata. +```bash +bin/pulsar initialize-cluster-metadata \ + --cluster cluster-a \ + --zookeeper zookeeper:2181 \ + --configuration-store zookeeper:2181 \ + --web-service-url http://broker:8080 \ + --broker-service-url pulsar://broker:6650 \ +``` +## Start broker and bookie services +Start the broker and bookie services in the Docker container by running the following commands. ### Start broker service 1. Open the broker container -``` +```bash docker exec -it broker bash ``` -2. Start the broker service +2. copy configuration from env +```bash +bin/apply-config-from-env.py conf/broker.conf ``` +3. Start the broker service +```bash bin/pulsar-daemon start broker ``` ### Start bookie service 1. Open the bookie container -``` +```bash docker exec -it bookie bash ``` -2. Start the bookie service +2. copy configuration from env +```bash +bin/apply-config-from-env.py conf/bookkeeper.conf ``` +3. Start the bookie service +```bash bin/pulsar-daemon start bookie ``` +## Deploy a cluster by docker-compose.yaml +The following is a docker-compose.yaml file which can help you to deploy the Pulsar cluster quickly. + +* Create a Pulsar cluster by docker-compose. +```bash +docker-compose up -d +``` +* Destroy a Pulsar cluster by docker-compose. +```bash +docker-compose down +``` + +```yaml +version: '3' +networks: + pulsar: + driver: bridge +services: + # Start zookeeper + zookeeper: + image: apachepulsar/pulsar:latest + container_name: zookeeper + restart: on-failure + networks: + - pulsar + volumes: + - ./data/zookeeper:/pulsar/data/zookeeper + environment: + - metadataStoreUrl=zk:zookeeper:2181 + command: > + bash -c "bin/apply-config-from-env.py conf/zookeeper.conf && \ + bin/generate-zookeeper-config.sh conf/zookeeper.conf && \ + exec bin/pulsar zookeeper" + healthcheck: + test: ["CMD", "bin/pulsar-zookeeper-ruok.sh"] + interval: 10s + timeout: 5s + retries: 30 + + # Init cluster metadata + pulsar-init: + container_name: pulsar-init + hostname: pulsar-init + image: apachepulsar/pulsar:latest + networks: + - pulsar + command: > + bin/pulsar initialize-cluster-metadata \ + --cluster cluster-a \ + --zookeeper zookeeper:2181 \ + --configuration-store zookeeper:2181 \ + --web-service-url http://broker:8080 \ + --broker-service-url pulsar://broker:6650 + depends_on: + zookeeper: + condition: service_healthy + + # Start bookie + bookie: + image: apachepulsar/pulsar:latest + container_name: bookie + restart: on-failure + networks: + - pulsar + environment: + - clusterName=cluster-a + - zkServers=zookeeper:2181 + - metadataServiceUri=metadata-store:zk:zookeeper:2181 + depends_on: + zookeeper: + condition: service_healthy + pulsar-init: + condition: service_completed_successfully + # Map the local directory to the container to avoid bookie startup failure due to insufficient container disks. + volumes: + - ./data/bookkeeper:/pulsar/data/bookkeeper + command: bash -c "bin/apply-config-from-env.py conf/bookkeeper.conf + && exec bin/pulsar bookie" + + # Start broker + broker: + image: apachepulsar/pulsar:latest + container_name: broker + hostname: broker + restart: on-failure + networks: + - pulsar + environment: + - metadataStoreUrl=zk:zookeeper:2181 + - zookeeperServers=zookeeper:2181 + - clusterName=cluster-a + - managedLedgerDefaultEnsembleSize=1 + - managedLedgerDefaultWriteQuorum=1 + - managedLedgerDefaultAckQuorum=1 + - advertisedAddress=broker + - advertisedListeners=external:pulsar://127.0.0.1:6650 + depends_on: + zookeeper: + condition: service_healthy + bookie: + condition: service_started + ports: + - "6650:6650" + - "8080:8080" + command: bash -c "bin/apply-config-from-env.py conf/broker.conf + && exec bin/pulsar broker" +``` + + From f9827b07eff6eb95c490f50f36bc8254a32dec77 Mon Sep 17 00:00:00 2001 From: liangyepianzhou Date: Mon, 10 Oct 2022 16:20:59 +0800 Subject: [PATCH 4/5] optimize and add docker-compose --- site2/docs/deploy-docker.md | 329 +++++++++++++++--------------------- 1 file changed, 133 insertions(+), 196 deletions(-) diff --git a/site2/docs/deploy-docker.md b/site2/docs/deploy-docker.md index ad3184d7c61cd..2b8e755106f9e 100644 --- a/site2/docs/deploy-docker.md +++ b/site2/docs/deploy-docker.md @@ -2,226 +2,163 @@ id: deploy-docker title: Deploy a cluster on Docker sidebar_label: "Docker" ---- -## Deploy a cluster on Docker -To deploy a Pulsar cluster on Docker, you need to complete the following steps: -1. Pull the Pulsar Docker image. -2. Create the zookeeper, bookie, broker container by the image -3. Modify the broker.conf and bookkeeper.conf -4. Create ZooKeeper, bookie, and broker containers. -5. Start ZooKeeper and initialize cluster metadata. -6. Start brokers and bookies. - -## Prepare -To run Pulsar on Docker, you need to create a container for each Pulsar component: ZooKeeper, BookKeeper and broker. You can pull the images of ZooKeeper and BookKeeper separately on Docker Hub, and pull a Pulsar image for the broker. You can also pull only one Pulsar image and create three containers with this image. This tutorial takes the second option as an example. - -## Pull a Pulsar image -You can pull a Pulsar image from Docker Hub with the following command. If you want to use some connectors, you can use apachepulsar/pulsar-all there. +— +You can use two kinds of methods to deploy a Pulsar cluster on Docker. +The first uses Docker commands, while the second uses a `docker-compose.yaml` file. +## Deploy a cluster using Docker commands +To deploy a Pulsar cluster on Docker, you need to complete the following steps: +Pull a Pulsar Docker image. +Create a network. +Create and start the ZooKeeper, bookie, and broker containers. +### Pull a Pulsar image +To run Pulsar on Docker, you need to create a container for each Pulsar component: ZooKeeper, bookie, and the broker. You can pull the images of ZooKeeper and bookie separately on Docker Hub, and pull the Pulsar image for the broker. You can also pull only one Pulsar image and create three containers with this image. This tutorial takes the second option as an example. +You can pull a Pulsar image from Docker Hub with the following command. If you do not want to use some connectors, you can use `apachepulsar/pulsar:latest` there. ```java docker pull apachepulsar/pulsar-all:latest ``` -## Create containers -Create a ZooKeeper container -```bash -docker run -d --privileged=true -u=root -p 2181:2181 -e metadataStoreUrl=zk:zookeeper:2181 -e cluster-name=cluster-a -e managedLedgerDefaultEnsembleSize=1 -e managedLedgerDefaultWriteQuorum=1 -e managedLedgerDefaultAckQuorum=1 --name zookeeper apachepulsar/pulsar-all:latest /bin/bash -``` - -* Create broker container - -```bash -docker run -d --privileged=true -u=root -p 6650:6650 -p 8080:8080 -e metadataStoreUrl=zk:zookeeper:2181 -e zookeeperServers=zookeeper:2181 -e clusterName=cluster-a -e managedLedgerDefaultEnsembleSize=1 -e managedLedgerDefaultWriteQuorum=1 -e managedLedgerDefaultAckQuorum=1 --name broker apachepulsar/pulsar-all:latest /bin/bash -``` - -* Create a bookie container - -```bash -docker run -d --privileged=true -u=root -e clusterName=cluster-a -e zkServers=zookeeper:2181 -e metadataServiceUri=metadata-store:zk:zookeeper:2181 --name bookie apachepulsar/pulsar-all:latest /bin/bash -``` -## Create the network -To deploy a Pulsar cluster on Docker, you need to create a network and connect the containers of ZooKeeper, BookKeeper and broker to this network. The following command creates the network pulsar: +### Create a network +To deploy a Pulsar cluster on Docker, you need to create a network and connect the containers of ZooKeeper, bookie, and broker to this network. +Use the following command to create the network `pulsar`: ```bash docker network create pulsar ``` -Connect the containers of ZooKeeper, BookKeeper and broker to the pulsar network with the following commands. +### Create and start containers +#### Create a ZooKeeper container +Create a ZooKeeper container and start the ZooKeeper service. ```bash -docker network connect pulsar zookeeper -``` -```bash -docker network connect pulsar bookkeeper +docker run -d -p 2181:2181 --net=pulsar -e metadataStoreUrl=zk:zookeeper:2181 -e cluster-name=cluster-a -e managedLedgerDefaultEnsembleSize=1 -e managedLedgerDefaultWriteQuorum=1 -e managedLedgerDefaultAckQuorum=1 -v $(pwd)/data/zookeeper:/pulsar/data/zookeeper --name zookeeper --hostname zookeeper apachepulsar/pulsar-all:latest bash -c "bin/apply-config-from-env.py conf/zookeeper.conf && bin/generate-zookeeper-config.sh conf/zookeeper.conf && exec bin/pulsar zookeeper" ``` +#### Initialize the cluster metadata +After creating the ZooKeeper container successfully, you can use the following command to initialize the cluster metadata. ```bash -docker network connect pulsar broker +docker run --net=pulsar --name initialize-pulsar-cluster-metadata apachepulsar/pulsar-all:latest bash -c "bin/pulsar initialize-cluster-metadata \ +--cluster cluster-a \ +--zookeeper zookeeper:2181 \ +--configuration-store zookeeper:2181 \ +--web-service-url http://broker:8080 \ +--broker-service-url pulsar://broker:6650" ``` -## Start ZooKeeper and initialize cluster metadata -Start the ZooKeeper service in the Docker container by running the following commands. -1. Open the ZooKeeper container. -```bash -docker exec -it zookeeper bash -``` -2. copy configuration from env -```bash -bin/apply-config-from-env.py conf/zookeeper.conf -``` +#### Create a bookie container +Create a bookie container and start the bookie service. ```bash -bin/generate-zookeeper-config.sh conf/zookeeper.conf +docker run -d -e clusterName=cluster-a -e zkServers=zookeeper:2181 --net=pulsar -e metadataServiceUri=metadata-store:zk:zookeeper:2181 -v $(pwd)/data/bookkeeper:/pulsar/data/bookkeeper --name bookie --hostname bookie apachepulsar/pulsar-all:latest bash -c "bin/apply-config-from-env.py conf/bookkeeper.conf && exec bin/pulsar bookie" ``` - -3. Start the ZooKeeper service. +#### Create a broker container +Create a broker container and start the broker service. ```bash -bin/pulsar-daemon start zookeeper +docker run -d -p 6650:6650 -p 8080:8080 --net=pulsar -e metadataStoreUrl=zk:zookeeper:2181 -e zookeeperServers=zookeeper:2181 -e clusterName=cluster-a -e managedLedgerDefaultEnsembleSize=1 -e managedLedgerDefaultWriteQuorum=1 -e managedLedgerDefaultAckQuorum=1 --name broker --hostname broker apachepulsar/pulsar-all:latest bash -c "bin/apply-config-from-env.py conf/broker.conf && exec bin/pulsar broker" ``` -4. Initialize the cluster metadata. -```bash -bin/pulsar initialize-cluster-metadata \ - --cluster cluster-a \ - --zookeeper zookeeper:2181 \ - --configuration-store zookeeper:2181 \ - --web-service-url http://broker:8080 \ - --broker-service-url pulsar://broker:6650 \ -``` -## Start broker and bookie services -Start the broker and bookie services in the Docker container by running the following commands. -### Start broker service -1. Open the broker container -```bash -docker exec -it broker bash -``` -2. copy configuration from env -```bash -bin/apply-config-from-env.py conf/broker.conf -``` -3. Start the broker service -```bash -bin/pulsar-daemon start broker -``` - -### Start bookie service -1. Open the bookie container -```bash -docker exec -it bookie bash -``` -2. copy configuration from env -```bash -bin/apply-config-from-env.py conf/bookkeeper.conf -``` -3. Start the bookie service -```bash -bin/pulsar-daemon start bookie -``` - -## Deploy a cluster by docker-compose.yaml -The following is a docker-compose.yaml file which can help you to deploy the Pulsar cluster quickly. - -* Create a Pulsar cluster by docker-compose. -```bash -docker-compose up -d -``` -* Destroy a Pulsar cluster by docker-compose. -```bash -docker-compose down -``` +## Deploy a cluster by using `docker-compose.yaml` +Use the following template to create a `docker-compose.yaml` file to deploy a Pulsar cluster quickly. And you can modify or add the configurations in the `environment` section. ```yaml version: '3' networks: - pulsar: - driver: bridge +pulsar: + driver: bridge services: - # Start zookeeper - zookeeper: - image: apachepulsar/pulsar:latest - container_name: zookeeper - restart: on-failure - networks: - - pulsar - volumes: - - ./data/zookeeper:/pulsar/data/zookeeper - environment: - - metadataStoreUrl=zk:zookeeper:2181 - command: > - bash -c "bin/apply-config-from-env.py conf/zookeeper.conf && \ - bin/generate-zookeeper-config.sh conf/zookeeper.conf && \ - exec bin/pulsar zookeeper" - healthcheck: - test: ["CMD", "bin/pulsar-zookeeper-ruok.sh"] - interval: 10s - timeout: 5s - retries: 30 - - # Init cluster metadata - pulsar-init: - container_name: pulsar-init - hostname: pulsar-init - image: apachepulsar/pulsar:latest - networks: - - pulsar - command: > - bin/pulsar initialize-cluster-metadata \ - --cluster cluster-a \ - --zookeeper zookeeper:2181 \ - --configuration-store zookeeper:2181 \ - --web-service-url http://broker:8080 \ - --broker-service-url pulsar://broker:6650 - depends_on: - zookeeper: - condition: service_healthy - - # Start bookie - bookie: - image: apachepulsar/pulsar:latest - container_name: bookie - restart: on-failure - networks: - - pulsar - environment: - - clusterName=cluster-a - - zkServers=zookeeper:2181 - - metadataServiceUri=metadata-store:zk:zookeeper:2181 - depends_on: - zookeeper: - condition: service_healthy - pulsar-init: - condition: service_completed_successfully - # Map the local directory to the container to avoid bookie startup failure due to insufficient container disks. - volumes: - - ./data/bookkeeper:/pulsar/data/bookkeeper - command: bash -c "bin/apply-config-from-env.py conf/bookkeeper.conf - && exec bin/pulsar bookie" - - # Start broker - broker: - image: apachepulsar/pulsar:latest - container_name: broker - hostname: broker - restart: on-failure - networks: - - pulsar - environment: - - metadataStoreUrl=zk:zookeeper:2181 - - zookeeperServers=zookeeper:2181 - - clusterName=cluster-a - - managedLedgerDefaultEnsembleSize=1 - - managedLedgerDefaultWriteQuorum=1 - - managedLedgerDefaultAckQuorum=1 - - advertisedAddress=broker - - advertisedListeners=external:pulsar://127.0.0.1:6650 - depends_on: - zookeeper: - condition: service_healthy - bookie: - condition: service_started - ports: - - "6650:6650" - - "8080:8080" - command: bash -c "bin/apply-config-from-env.py conf/broker.conf - && exec bin/pulsar broker" +# Start zookeeper +zookeeper: + image: apachepulsar/pulsar:latest + container_name: zookeeper + restart: on-failure + networks: + - pulsar + volumes: + - ./data/zookeeper:/pulsar/data/zookeeper + environment: + - metadataStoreUrl=zk:zookeeper:2181 + command: > + bash -c "bin/apply-config-from-env.py conf/zookeeper.conf && \ + bin/generate-zookeeper-config.sh conf/zookeeper.conf && \ + exec bin/pulsar zookeeper" + healthcheck: + test: ["CMD", "bin/pulsar-zookeeper-ruok.sh"] + interval: 10s + timeout: 5s + retries: 30 + +# Init cluster metadata +pulsar-init: + container_name: pulsar-init + hostname: pulsar-init + image: apachepulsar/pulsar:latest + networks: + - pulsar + command: > + bin/pulsar initialize-cluster-metadata \ + --cluster cluster-a \ + --zookeeper zookeeper:2181 \ + --configuration-store zookeeper:2181 \ + --web-service-url http://broker:8080 \ + --broker-service-url pulsar://broker:6650 + depends_on: + zookeeper: + condition: service_healthy + +# Start bookie +bookie: + image: apachepulsar/pulsar:latest + container_name: bookie + restart: on-failure + networks: + - pulsar + environment: + - clusterName=cluster-a + - zkServers=zookeeper:2181 + - metadataServiceUri=metadata-store:zk:zookeeper:2181 + depends_on: + zookeeper: + condition: service_healthy + pulsar-init: + condition: service_completed_successfully + # Map the local directory to the container to avoid bookie startup failure due to insufficient container disks. + volumes: + - ./data/bookkeeper:/pulsar/data/bookkeeper + command: bash -c "bin/apply-config-from-env.py conf/bookkeeper.conf + && exec bin/pulsar bookie" + +# Start broker +broker: + image: apachepulsar/pulsar:latest + container_name: broker + hostname: broker + restart: on-failure + networks: + - pulsar + environment: + - metadataStoreUrl=zk:zookeeper:2181 + - zookeeperServers=zookeeper:2181 + - clusterName=cluster-a + - managedLedgerDefaultEnsembleSize=1 + - managedLedgerDefaultWriteQuorum=1 + - managedLedgerDefaultAckQuorum=1 + - advertisedAddress=broker + - advertisedListeners=external:pulsar://127.0.0.1:6650 + depends_on: + zookeeper: + condition: service_healthy + bookie: + condition: service_started + ports: + - "6650:6650" + - "8080:8080" + command: bash -c "bin/apply-config-from-env.py conf/broker.conf + && exec bin/pulsar broker" +``` + +To create a Pulsar cluster by using the `docker-compose.yaml` file, run the following command. +```bash +docker-compose up -d ``` - - +If you want to destroy the Pulsar cluster with all the containers, run the following command. It will also delete the network that the containers are connected to. +```bash +docker-compose down +``` From 165237affaef001b74003cad001ebed5cc9fcc79 Mon Sep 17 00:00:00 2001 From: liangyepianzhou Date: Mon, 10 Oct 2022 19:40:17 +0800 Subject: [PATCH 5/5] optimize --- site2/docs/deploy-docker.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/site2/docs/deploy-docker.md b/site2/docs/deploy-docker.md index 2b8e755106f9e..718eba80ab303 100644 --- a/site2/docs/deploy-docker.md +++ b/site2/docs/deploy-docker.md @@ -8,9 +8,9 @@ You can use two kinds of methods to deploy a Pulsar cluster on Docker. The first uses Docker commands, while the second uses a `docker-compose.yaml` file. ## Deploy a cluster using Docker commands To deploy a Pulsar cluster on Docker, you need to complete the following steps: -Pull a Pulsar Docker image. -Create a network. -Create and start the ZooKeeper, bookie, and broker containers. +1. Pull a Pulsar Docker image. +2. Create a network. +3. Create and start the ZooKeeper, bookie, and broker containers. ### Pull a Pulsar image To run Pulsar on Docker, you need to create a container for each Pulsar component: ZooKeeper, bookie, and the broker. You can pull the images of ZooKeeper and bookie separately on Docker Hub, and pull the Pulsar image for the broker. You can also pull only one Pulsar image and create three containers with this image. This tutorial takes the second option as an example. You can pull a Pulsar image from Docker Hub with the following command. If you do not want to use some connectors, you can use `apachepulsar/pulsar:latest` there.