fix: changes for hokusai dev/test. (#2889)

* fix: replace docker-compose extends with jinja template.

* fix: replace extends with jinja template, in test.yml.

* fix: re-arrange Docker build steps, to reduce delay in hokusai dev start.

* fix: build.yml can be removed now extends is not used.

* fix: update README for hokusai dev

* feat: use host network mode for docker compose services.

* fix: with development.yml using host network mode, mongo/es URL's can be taken from .env. update README accordingly.

* fix: change .env.example ELASTICSEARCH_URL to localhost.

Author: artsyjian

.env.example

@@ -9,7 +9,7 @@ SESSION_SECRET=p0s1tr0n
 ELASTICSEARCH_PORT=9200
 
 # Local development only for the below; refers to the key in docker-compose.yml
-ELASTICSEARCH_URL=http://elasticsearch:9200
+ELASTICSEARCH_URL=http://localhost:9200
 MONGOHQ_URL=mongodb://localhost:27017/positron
 
 ##

Dockerfile

@@ -18,17 +18,19 @@ RUN apk add --no-cache --quiet \
 COPY package.json yarn.lock ./
 RUN yarn install --frozen-lockfile && yarn cache clean
 
-# Copy application code
-COPY . ./
+# Update file/directory permissions
+RUN chown -R deploy:deploy ./
 
-# Ensure COMMIT_HASH is present
-RUN test -n "$COMMIT_HASH" && \
-  echo $COMMIT_HASH > COMMIT_HASH.txt && \
-  # Update file/directory permissions
-  chown -R deploy:deploy ./
+# Copy application code
+COPY --chown=deploy:deploy . ./
 
 # Switch to less-privileged user
 USER deploy
 
+# Ensure COMMIT_HASH is present
+# Run this step as late as possible b/c it busts docker cache.
+RUN test -n "$COMMIT_HASH" && \
+  echo $COMMIT_HASH > COMMIT_HASH.txt
+
 ENTRYPOINT ["/usr/bin/dumb-init", "--"]
 CMD ["yarn", "start"]

README.md

@@ -22,39 +22,35 @@
 
 [![Build Status](https://circleci.com/gh/artsy/positron/tree/master.svg?style=svg)](https://circleci.com/gh/artsy/positron/tree/master) [![codecov](https://codecov.io/gh/artsy/positron/branch/master/graph/badge.svg)](https://codecov.io/gh/artsy/positron)
 
-## Set-Up
+## Setup
 
-### Via Hokusai
-
-- Set up [Hokusai](https://github.com/artsy/README/blob/master/playbooks/hokusai.md#quickstart)
-- `git clone git@github.com:<your username>/positron.git && cd positron`
-- `COMMIT_HASH=$(git rev-parse --short HEAD) hokusai dev start`
-
-This starts a new Docker Compose stack that boots MongoDB, ElasticSearch and Positron. Changes made to source-code are _not_ automatically reloaded. To shut down, press `ctrl+c` or execute `hokusai dev stop`.
-
-### Manually
-
-- Install [NVM](https://github.com/creationix/nvm)
-- Install Node 12
-
-```
-nvm install 12
-nvm alias default 12
-```
+### Preparation
 
 - Fork Positron to your Github account in the Github UI.
+
 - Clone your repo locally (substitute your Github username).
 
 ```
 git clone git@github.com:craigspaeth/positron.git && cd positron
 ```
+- Setup [Hokusai](https://github.com/artsy/README/blob/master/playbooks/hokusai.md#quickstart)
 
 - Copy `.env.example` to `.env` in the root of the project and edit all `REPLACE` values with sensitive configuration obtained from `positron-staging`. Use the following command:
 
 ```
 hokusai staging env get | grep -E `cat .env.example | grep REPLACE | cut -f1 -d= | xargs | tr ' ' \|` | sed -e 's/:\ /=/g' | sed -e 's/ //g'
 ```
 
+### Installs (skip if you use hokusai dev, please see section below)
+
+- Install [NVM](https://github.com/creationix/nvm)
+- Install Node 12
+
+```
+nvm install 12
+nvm alias default 12
+```
+
 - Install node modules
 
 ```
@@ -80,10 +76,15 @@ brew install elasticsearch
 brew services start elasticsearch
 ```
 
-- Create a dummy channel
-  In order to write articles, you will need to be a member of a channel. If you are an Artsy dev, you will likely want to point your MONGOHQ_URL to the staging database. Because staging/production databases are protected on our VPC, it is required that you give user permissions to your AWS account and run the [Tunnelblick VPN](https://tunnelblick.net/) while connecting to the database. See details on [setting up a VPN connection here](https://github.com/artsy/infrastructure/blob/master/README.md#vpn).
+### Prepare database
+
+#### Using staging database
+
+In order to write articles, you will need to be a member of a channel. If you are an Artsy dev, you can point MONGOHQ_URL env to the staging database. Connecting to staging database requires VPN, please see details on [setting up a VPN connection here](https://github.com/artsy/infrastructure/blob/master/README.md#vpn).
+
+#### Using a local database
 
-  If using a local database, use these steps to backfill required data:
+With MongoDB running locally, follow these steps to create a dummy channel:
 
 1. Create a collection called `channels` in a `positron` db in your mongo database (You can use the mongo shell or a simple UI like Robomongo.)
 2. Add a document with the following fields:
@@ -96,23 +97,44 @@ brew services start elasticsearch
 }
 ```
 
-- Start the server
+If you are using Hokusai dev, start the stack as mentioned in subsequent section, edit the database as mentioned in this step, then restart the stack.
+
+
+### Start the server
+
+#### Using Yarn
 
 ```
 yarn start
 ```
 
-- Positron should now be running at [http://localhost:3005/](http://localhost:3005/), open a browser and navigate to it. That will redirect you to staging, login as an Artsy administrator and it will redirect you to `http://localhost:3005` logged into Writer. If you are an Artsy Admin pointed to the staging database, you should see the default partner gallery channel (David Zwirner).
+#### Using Hokusai Dev
 
-If you aren't an artsy admin you'll possibly get an Unauthorized page. You need to do one more mongo operation: edit the `users` collection and set your user's `channel_ids` to `[ ObjectId("<your_above_channel_id>") ]`. Once that's done you should be able to see the main writer interface.
+`COMMIT_HASH=$(git rev-parse --short HEAD) hokusai dev start`
+
+This starts a new Docker Compose stack that boots MongoDB, ElasticSearch and Positron. Changes made to source-code are _not_ automatically reloaded. To shut down, press `ctrl+c` or execute `hokusai dev stop`.
 
-- Run tests
+
+Positron should now be running at [http://localhost:3005/](http://localhost:3005/), open a browser and navigate to it. That will redirect you to staging, login as an Artsy administrator and it will redirect you to `http://localhost:3005` logged into Writer.
+
+If you are an Artsy Admin, you should see the default partner gallery channel (David Zwirner). If you aren't an artsy admin you'll possibly get an Unauthorized page. You need to do one more mongo operation: edit the `users` collection and set your user's `channel_ids` to `[ ObjectId("<your_above_channel_id>") ]`. Once that's done you should be able to see the main writer interface.
+
+##  Run tests
+
+### Using Yarn
 
 ```
 yarn test
 ```
 
-- Make sure you have mongo running in the background or most tests will not work.
+Make sure you have mongo running in the background or most tests will not work.
+
+### Using Hokusai
+
+```
+hokusai test
+```
+
 
 ## Debugging
 

hokusai/development.yml

@@ -1,30 +1,22 @@
 version: "2"
 services:
   positron:
+{% include 'templates/docker-compose-service.yml.j2' %}
     command: ["yarn", "dev"]
-    extends:
-      file: build.yml
-      service: positron
-    ports:
-      - 3005:3005
-    environment:
-      - ELASTICSEARCH_URL=http://positron-elasticsearch:9200
-      - MONGOHQ_URL=mongodb://positron-mongodb:27017/positron
     env_file: ../.env
     depends_on:
       - positron-mongodb
       - positron-elasticsearch
+    network_mode: "host"
   positron-elasticsearch:
     image: docker.elastic.co/elasticsearch/elasticsearch:5.6.16
     environment:
       - discovery.type=single-node
       - xpack.security.enabled=false
       - bootstrap.memory_lock=true
       - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
-    ports:
-      - "9200:9200"
+    network_mode: "host"
   positron-mongodb:
     image: mongo:4.0
-    ports:
-      - "27017:27017"
     command: ["--storageEngine=mmapv1", "--quiet", "--nojournal"]
+    network_mode: "host"

hokusai/build.yml …/templates/docker-compose-service.yml.j2hokusai/build.yml renamed to hokusai/templates/docker-compose-service.yml.j2 hokusai/build.yml renamed to hokusai/templates/docker-compose-service.yml.j2

@@ -1,6 +1,3 @@
-version: "2"
-services:
-  positron:
     build:
       context: ../
       args:

hokusai/test.yml

@@ -2,9 +2,7 @@ version: "2"
 services:
   positron:
     command: ["yarn", "test"]
-    extends:
-      file: build.yml
-      service: positron
+{% include 'templates/docker-compose-service.yml.j2' %}
     environment:
       - CI=true
       - ELASTICSEARCH_URL=http://positron-elasticsearch:9200