Skip to content

[fix][doc] Rewrite the steps of docker-deploy #17853

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
liangyepianzhou merged 5 commits into from
Oct 10, 2022
Merged

[fix][doc] Rewrite the steps of docker-deploy #17853

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9c4d44c
[fxi][doc] Rewrite the steps of docker-deploy
liangyepianzhou Sep 27, 2022
b538438
replace conf with env
liangyepianzhou Sep 29, 2022
12e751e
optimize the doc and add a docker-compose.yaml to deploy Pulsar cluster
liangyepianzhou Oct 2, 2022
f9827b0
optimize and add docker-compose
liangyepianzhou Oct 10, 2022
165237a
optimize
liangyepianzhou Oct 10, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
178 changes: 145 additions & 33 deletions site2/docs/deploy-docker.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,51 +2,163 @@
id: deploy-docker
title: Deploy a cluster on Docker
sidebar_label: "Docker"
---

To deploy a Pulsar cluster on Docker, complete the following steps:
1. Deploy a ZooKeeper cluster (optional)
2. Initialize cluster metadata
3. Deploy a BookKeeper cluster
4. Deploy one or more Pulsar brokers

## Prepare

To run Pulsar on Docker, you need to create a container for each Pulsar component: ZooKeeper, BookKeeper and broker. You can pull the images of ZooKeeper and BookKeeper separately on [Docker Hub](https://hub.docker.com/), and pull a [Pulsar image](https://hub.docker.com/r/apachepulsar/pulsar-all/tags) for the broker. You can also pull only one [Pulsar image](https://hub.docker.com/r/apachepulsar/pulsar-all/tags) and create three containers with this image. This tutorial takes the second option as an example.
—

@tisonkun tisonkun Oct 11, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an obvious syntax error that fails the site build.

Please be careful when reviewing...Always previewing your changes helps.

@liangyepianzhou liangyepianzhou Oct 11, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made a suggestion to modify the description template. I think we should guarantee correctness from the process and institutional should be, rather than expecting reviewers and authors always to be correct.

@tisonkun tisonkun Oct 11, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. The community runs by meritocracy so reviewers should be careful. It's not irreversible to make mistake like this and I'm not blaming you here. But send a notice that you should follow the process. Adding more description hints you to follow the process also but I believe few people really read the whole description.

@xiaogang-sn xiaogang-sn Oct 11, 2022

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't there be a sanity check CI job for any documentation change in the repo if full pipeline is too expensive?


You can use two kinds of methods to deploy a Pulsar cluster on Docker.
The first uses Docker commands, while the second uses a `docker-compose.yaml` file.
## Deploy a cluster using Docker commands
To deploy a Pulsar cluster on Docker, you need to complete the following steps:
1. Pull a Pulsar Docker image.
2. Create a network.
3. Create and start the ZooKeeper, bookie, and broker containers.
### Pull a Pulsar image
You can pull a Pulsar image from [Docker Hub](https://hub.docker.com/r/apachepulsar/pulsar-all/tags) with the following command.

```shell
To run Pulsar on Docker, you need to create a container for each Pulsar component: ZooKeeper, bookie, and the broker. You can pull the images of ZooKeeper and bookie separately on Docker Hub, and pull the Pulsar image for the broker. You can also pull only one Pulsar image and create three containers with this image. This tutorial takes the second option as an example.
You can pull a Pulsar image from Docker Hub with the following command. If you do not want to use some connectors, you can use `apachepulsar/pulsar:latest` there.
```java
docker pull apachepulsar/pulsar-all:latest
```
### Create a network
To deploy a Pulsar cluster on Docker, you need to create a network and connect the containers of ZooKeeper, bookie, and broker to this network.
Use the following command to create the network `pulsar`:
```bash
docker network create pulsar
```
Comment thread
liangyepianzhou marked this conversation as resolved.
### Create and start containers

### Create three containers
Create containers for ZooKeeper, BookKeeper and broker. In this example, they are named as `zookeeper`, `bookkeeper` and `broker` respectively. You can name them as you want with the `--name` flag. By default, the container names are created randomly.

```shell
docker run -it --name bookkeeper apachepulsar/pulsar-all:latest /bin/bash
docker run -it --name zookeeper apachepulsar/pulsar-all:latest /bin/bash
docker run -it --name broker apachepulsar/pulsar-all:latest /bin/bash
#### Create a ZooKeeper container
Create a ZooKeeper container and start the ZooKeeper service.
```bash
docker run -d -p 2181:2181 --net=pulsar -e metadataStoreUrl=zk:zookeeper:2181 -e cluster-name=cluster-a -e managedLedgerDefaultEnsembleSize=1 -e managedLedgerDefaultWriteQuorum=1 -e managedLedgerDefaultAckQuorum=1 -v $(pwd)/data/zookeeper:/pulsar/data/zookeeper --name zookeeper --hostname zookeeper apachepulsar/pulsar-all:latest bash -c "bin/apply-config-from-env.py conf/zookeeper.conf && bin/generate-zookeeper-config.sh conf/zookeeper.conf && exec bin/pulsar zookeeper"
```
#### Initialize the cluster metadata
After creating the ZooKeeper container successfully, you can use the following command to initialize the cluster metadata.
```bash
docker run --net=pulsar --name initialize-pulsar-cluster-metadata apachepulsar/pulsar-all:latest bash -c "bin/pulsar initialize-cluster-metadata \
--cluster cluster-a \
--zookeeper zookeeper:2181 \
--configuration-store zookeeper:2181 \
--web-service-url http://broker:8080 \
--broker-service-url pulsar://broker:6650"
```

### Create a network
To deploy a Pulsar cluster on Docker, you need to create a `network` and connect the containers of ZooKeeper, BookKeeper and broker to this network. The following command creates the network `pulsar`:

```shell
docker network create pulsar
#### Create a bookie container
Create a bookie container and start the bookie service.

```bash
docker run -d -e clusterName=cluster-a -e zkServers=zookeeper:2181 --net=pulsar -e metadataServiceUri=metadata-store:zk:zookeeper:2181 -v $(pwd)/data/bookkeeper:/pulsar/data/bookkeeper --name bookie --hostname bookie apachepulsar/pulsar-all:latest bash -c "bin/apply-config-from-env.py conf/bookkeeper.conf && exec bin/pulsar bookie"
```
#### Create a broker container
Create a broker container and start the broker service.
```bash
docker run -d -p 6650:6650 -p 8080:8080 --net=pulsar -e metadataStoreUrl=zk:zookeeper:2181 -e zookeeperServers=zookeeper:2181 -e clusterName=cluster-a -e managedLedgerDefaultEnsembleSize=1 -e managedLedgerDefaultWriteQuorum=1 -e managedLedgerDefaultAckQuorum=1 --name broker --hostname broker apachepulsar/pulsar-all:latest bash -c "bin/apply-config-from-env.py conf/broker.conf && exec bin/pulsar broker"
```

### Connect containers to network
Connect the containers of ZooKeeper, BookKeeper and broker to the `pulsar` network with the following commands.
## Deploy a cluster by using `docker-compose.yaml`
Use the following template to create a `docker-compose.yaml` file to deploy a Pulsar cluster quickly. And you can modify or add the configurations in the `environment` section.

```shell
docker network connect pulsar zookeeper
docker network connect pulsar bookkeeper
docker network connect pulsar broker
```yaml
version: '3'
networks:
pulsar:
driver: bridge
services:
# Start zookeeper
zookeeper:
image: apachepulsar/pulsar:latest
container_name: zookeeper
restart: on-failure
networks:
- pulsar
volumes:
- ./data/zookeeper:/pulsar/data/zookeeper
environment:
- metadataStoreUrl=zk:zookeeper:2181
command: >
bash -c "bin/apply-config-from-env.py conf/zookeeper.conf && \
bin/generate-zookeeper-config.sh conf/zookeeper.conf && \
exec bin/pulsar zookeeper"
healthcheck:
test: ["CMD", "bin/pulsar-zookeeper-ruok.sh"]
interval: 10s
timeout: 5s
retries: 30

# Init cluster metadata
pulsar-init:
container_name: pulsar-init
hostname: pulsar-init
image: apachepulsar/pulsar:latest
networks:
- pulsar
command: >
bin/pulsar initialize-cluster-metadata \
--cluster cluster-a \
--zookeeper zookeeper:2181 \
--configuration-store zookeeper:2181 \
--web-service-url http://broker:8080 \
--broker-service-url pulsar://broker:6650
depends_on:
zookeeper:
condition: service_healthy

# Start bookie
bookie:
image: apachepulsar/pulsar:latest
container_name: bookie
restart: on-failure
networks:
- pulsar
environment:
- clusterName=cluster-a
- zkServers=zookeeper:2181
- metadataServiceUri=metadata-store:zk:zookeeper:2181
depends_on:
zookeeper:
condition: service_healthy
pulsar-init:
condition: service_completed_successfully
# Map the local directory to the container to avoid bookie startup failure due to insufficient container disks.
volumes:
- ./data/bookkeeper:/pulsar/data/bookkeeper
command: bash -c "bin/apply-config-from-env.py conf/bookkeeper.conf
&& exec bin/pulsar bookie"

# Start broker
broker:
image: apachepulsar/pulsar:latest
container_name: broker
hostname: broker
restart: on-failure
networks:
- pulsar
environment:
- metadataStoreUrl=zk:zookeeper:2181
- zookeeperServers=zookeeper:2181
- clusterName=cluster-a
- managedLedgerDefaultEnsembleSize=1
- managedLedgerDefaultWriteQuorum=1
- managedLedgerDefaultAckQuorum=1
- advertisedAddress=broker
- advertisedListeners=external:pulsar://127.0.0.1:6650
depends_on:
zookeeper:
condition: service_healthy
bookie:
condition: service_started
ports:
- "6650:6650"
- "8080:8080"
command: bash -c "bin/apply-config-from-env.py conf/broker.conf
&& exec bin/pulsar broker"
```

To check whether the containers are successfully connected to the network, enter the `docker network inspect pulsar` command.
To create a Pulsar cluster by using the `docker-compose.yaml` file, run the following command.
```bash
docker-compose up -d
```

For detailed information about how to deploy ZooKeeper cluster, BookKeeper cluster, brokers, see [deploy a cluster on bare metal](deploy-bare-metal.md).
If you want to destroy the Pulsar cluster with all the containers, run the following command. It will also delete the network that the containers are connected to.
```bash
docker-compose down
```