Skip to content

Allow to configure Terraform State #2438

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sachilles merged 4 commits into from
Apr 22, 2022
Merged

Allow to configure Terraform State #2438

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
4752be9
Allow to configure terraform state feature
peter-jozsa Apr 11, 2022
7374743
fix wrong footer comment of GCS related terraform state configuration
peter-jozsa Apr 11, 2022
82e88ff
create terraform state directory based on new configuration variables…
peter-jozsa Apr 11, 2022
950bb3d
Update README.md
sachilles Apr 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 57 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1132,7 +1132,7 @@ AWS Endpoint like `http://127.0.0.1:9000`. Defaults to `$AWS_ENDPOINT`

##### `GITLAB_LFS_OBJECT_STORE_CONNECTION_AWS_PATH_STYLE`

Changes AWS Path Style to 'host/bucket_name/object' instead of 'bucket_name.host/object'. Defaults to `AWS_PATH_STYLE`
Changes AWS Path Style to 'host/bucket_name/object' instead of 'bucket_name.host/object'. Defaults to `$AWS_PATH_STYLE`

##### `GITLAB_LFS_OBJECT_STORE_CONNECTION_AWS_SIGNATURE_VERSION`

Expand Down Expand Up @@ -1218,6 +1218,62 @@ Google service account. Defaults to `$GITLAB_OBJECT_STORE_CONNECTION_GOOGLE_CLIE

Default Google key file. Defaults to `$GITLAB_OBJECT_STORE_CONNECTION_GOOGLE_JSON_KEY_LOCATION` (`/gcs/key.json`)

##### `GITLAB_TERRAFORM_STATE_ENABLED`

Enable/Disable Terraform State support. Defaults to `true`.

##### `GITLAB_TERRAFORM_STATE_STORAGE_PATH`

Directory to store the terraform state data. Defaults to `$GITLAB_SHARED_DIR/terraform_state`

##### `GITLAB_TERRAFORM_STATE_OBJECT_STORE_ENABLED`

Enables Object Store for Terraform state that will be remote stored. Defaults to `false`

##### `GITLAB_TERRAFORM_STATE_OBJECT_STORE_REMOTE_DIRECTORY`

Bucket name to store the Terraform state. Defaults to `terraform_state`

##### `GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_PROVIDER`

Connection Provider for the Object Store (AWS or Google). Defaults to $GITLAB_OBJECT_STORE_CONNECTION_PROVIDER (i.e. AWS).

##### `GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_ACCESS_KEY_ID`

AWS Access Key ID for the Bucket. Defaults to `$AWS_ACCESS_KEY_ID`

##### `GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_SECRET_ACCESS_KEY`

AWS Secret Access Key. Defaults to `$AWS_SECRET_ACCESS_KEY`

##### `GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_REGION`

AWS Region. Defaults to `$AWS_REGION`

##### `GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_HOST`

Configure this for an compatible AWS host like minio. Defaults to `$AWS_HOST`

##### `GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_ENDPOINT`

AWS Endpoint like `http://127.0.0.1:9000`. Defaults to `$AWS_ENDPOINT`

##### `GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_PATH_STYLE`

Changes AWS Path Style to 'host/bucket_name/object' instead of 'bucket_name.host/object'. Defaults to `AWS_PATH_STYLE`

@sachilles sachilles Apr 21, 2022

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please change AWS_PATH_STYLE to $AWS_PATH_STYLE


##### `GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_GOOGLE_PROJECT`

Google project. Defaults to `$GITLAB_OBJECT_STORE_CONNECTION_GOOGLE_PROJECT`

##### `GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_GOOGLE_CLIENT_EMAIL`

Google service account. Defaults to `$GITLAB_OBJECT_STORE_CONNECTION_GOOGLE_CLIENT_EMAIL`

##### `GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_GOOGLE_JSON_KEY_LOCATION`

Default Google key file. Defaults to `$GITLAB_OBJECT_STORE_CONNECTION_GOOGLE_JSON_KEY_LOCATION` (`/gcs/key.json`)

##### `GITLAB_UPLOADS_STORAGE_PATH`

The location where uploads objects are stored. Defaults to `$GITLAB_SHARED_DIR/public`.
Expand Down
31 changes: 19 additions & 12 deletions assets/runtime/config/gitlabhq/gitlab.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -371,21 +371,28 @@ production: &base

## Terraform state
terraform_state:
enabled: true
enabled: {{GITLAB_TERRAFORM_STATE_ENABLED}}
# The location where Terraform state files are stored (default: shared/terraform_state).
# storage_path: shared/terraform_state
storage_path: {{GITLAB_TERRAFORM_STATE_STORAGE_PATH}}
object_store:
enabled: false
remote_directory: terraform_state # The bucket name
enabled: {{GITLAB_TERRAFORM_STATE_OBJECT_STORE_ENABLED}}
remote_directory: {{GITLAB_TERRAFORM_STATE_OBJECT_STORE_REMOTE_DIRECTORY}} # The bucket name
connection:
provider: AWS
aws_access_key_id: AWS_ACCESS_KEY_ID
aws_secret_access_key: AWS_SECRET_ACCESS_KEY
region: us-east-1
# host: 'localhost' # default: s3.amazonaws.com
# endpoint: 'http://127.0.0.1:9000' # default: nil
# aws_signature_version: 4 # For creation of signed URLs. Set to 2 if provider does not support v4.
# path_style: true # Use 'host/bucket_name/object' instead of 'bucket_name.host/object'
provider: {{GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_PROVIDER}}
#start-terraform_state-aws
aws_access_key_id: {{GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_ACCESS_KEY_ID}}
aws_secret_access_key: {{GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_SECRET_ACCESS_KEY}}
region: {{GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_REGION}}
host: '{{GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_HOST}}' # default: s3.amazonaws.com
endpoint: '{{GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_ENDPOINT}}' # default: nil
aws_signature_version: {{GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_SIGNATURE_VERSION}} # For creation of signed URLs. Set to 2 if provider does not support v4.
path_style: {{GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_PATH_STYLE}} # Use 'host/bucket_name/object' instead of 'bucket_name.host/object'
#end-terraform_state-aws
#start-terraform_state-gcs
google_project: {{GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_GOOGLE_PROJECT}}
google_client_email: {{GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_GOOGLE_CLIENT_EMAIL}}
google_json_key_location: {{GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_GOOGLE_JSON_KEY_LOCATION}}
#end-terraform_state-gcs

## GitLab Pages
pages:
Expand Down
22 changes: 22 additions & 0 deletions assets/runtime/env-defaults
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,28 @@ GITLAB_PACKAGES_OBJECT_STORE_CONNECTION_GOOGLE_PROJECT=${GITLAB_PACKAGES_OBJECT_
GITLAB_PACKAGES_OBJECT_STORE_CONNECTION_GOOGLE_CLIENT_EMAIL=${GITLAB_PACKAGES_OBJECT_STORE_CONNECTION_GOOGLE_CLIENT_EMAIL:-$GITLAB_OBJECT_STORE_CONNECTION_GOOGLE_CLIENT_EMAIL}
GITLAB_PACKAGES_OBJECT_STORE_CONNECTION_GOOGLE_JSON_KEY_LOCATION=${GITLAB_PACKAGES_OBJECT_STORE_CONNECTION_GOOGLE_JSON_KEY_LOCATION:-$GITLAB_OBJECT_STORE_CONNECTION_GOOGLE_JSON_KEY_LOCATION}

## TERRAFORM STATE
GITLAB_TERRAFORM_STATE_ENABLED=${GITLAB_TERRAFORM_STATE_ENABLED:-true}
GITLAB_TERRAFORM_STATE_STORAGE_PATH="${GITLAB_TERRAFORM_STATE_STORAGE_PATH:-$GITLAB_SHARED_DIR/terraform_state}"

GITLAB_TERRAFORM_STATE_OBJECT_STORE_ENABLED=${GITLAB_TERRAFORM_STATE_OBJECT_STORE_ENABLED:-false}
GITLAB_TERRAFORM_STATE_OBJECT_STORE_REMOTE_DIRECTORY=${GITLAB_PACKAGES_OBJECT_STORE_REMOTE_DIRECTORY:-terraform_state}
GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_PROVIDER=${GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_PROVIDER:-$GITLAB_OBJECT_STORE_CONNECTION_PROVIDER}

# TERRAFORM STATE:AWS
GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_ACCESS_KEY_ID=${GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_ACCESS_KEY_ID:-$AWS_ACCESS_KEY_ID}
GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_SECRET_ACCESS_KEY=${GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_SECRET_ACCESS_KEY:-$AWS_SECRET_ACCESS_KEY}
GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_REGION=${GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_REGION:-$AWS_REGION}
GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_HOST=${GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_HOST:-$AWS_HOST}
GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_ENDPOINT=${GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_ENDPOINT:-$AWS_ENDPOINT}
GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_PATH_STYLE=${GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_PATH_STYLE:-$AWS_PATH_STYLE}
GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_SIGNATURE_VERSION=${GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_SIGNATURE_VERSION:-$AWS_SIGNATURE_VERSION}

# TERRAFORM STATE:Google
GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_GOOGLE_PROJECT=${GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_GOOGLE_PROJECT:-$GITLAB_OBJECT_STORE_CONNECTION_GOOGLE_PROJECT}
GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_GOOGLE_CLIENT_EMAIL=${GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_GOOGLE_CLIENT_EMAIL:-$GITLAB_OBJECT_STORE_CONNECTION_GOOGLE_CLIENT_EMAIL}
GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_GOOGLE_JSON_KEY_LOCATION=${GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_GOOGLE_JSON_KEY_LOCATION:-$GITLAB_OBJECT_STORE_CONNECTION_GOOGLE_JSON_KEY_LOCATION}

## Cron Jobs
GITLAB_PIPELINE_SCHEDULE_WORKER_CRON=${GITLAB_PIPELINE_SCHEDULE_WORKER_CRON:-"19 * * * *"}

Expand Down
52 changes: 47 additions & 5 deletions assets/runtime/functions
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1068,6 +1068,47 @@ gitlab_configure_packages() {
GITLAB_PACKAGES_DIR
}

gitlab_configure_terraform_state() {
if [[ ${GITLAB_TERRAFORM_STATE_OBJECT_STORE_ENABLED} == true ]]; then
echo "Configuring gitlab::terraform_state:object_store"

if [[ "${GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_PROVIDER}" == "Google" ]]; then
echo " -> Google TERRAFORM STATE provider selected removing aws config"
exec_as_git sed -i "/#start-terraform_state-aws/,/#end-terraform_state-aws/d" ${GITLAB_CONFIG}
exec_as_git sed -i "/#start-terraform_state-gcs/d" ${GITLAB_CONFIG}
exec_as_git sed -i "/#end-terraform_state-gcs/d" ${GITLAB_CONFIG}
fi
if [[ "${GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_PROVIDER}" == "AWS" ]]; then
echo " -> AWS TERRAFORM STATE provider selected removing Google config"
exec_as_git sed -i "/#start-terraform_state-gcs/,/#end-terraform_state-gcs/d" ${GITLAB_CONFIG}
exec_as_git sed -i "/#start-terraform_state-aws/d" ${GITLAB_CONFIG}
exec_as_git sed -i "/#end-terraform_state-aws/d" ${GITLAB_CONFIG}
fi

update_template ${GITLAB_CONFIG} \
GITLAB_TERRAFORM_STATE_OBJECT_STORE_ENABLED \
GITLAB_TERRAFORM_STATE_OBJECT_STORE_REMOTE_DIRECTORY \
GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_PROVIDER \
GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_ACCESS_KEY_ID \
GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_SECRET_ACCESS_KEY \
GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_REGION \
GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_HOST \
GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_ENDPOINT \
GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_PATH_STYLE \
GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_AWS_SIGNATURE_VERSION \
GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_GOOGLE_PROJECT \
GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_GOOGLE_CLIENT_EMAIL \
GITLAB_TERRAFORM_STATE_OBJECT_STORE_CONNECTION_GOOGLE_JSON_KEY_LOCATION
else
exec_as_git sed -i -e "/storage_path: {{GITLAB_TERRAFORM_STATE_STORAGE_PATH}}/{n;N;N;N;N;N;N;N;N;N;N;N;N;N;N;N;N;N;d;}" ${GITLAB_CONFIG}
fi

echo "Configuring gitlab::terraform_state..."
update_template ${GITLAB_CONFIG} \
GITLAB_TERRAFORM_STATE_ENABLED \
GITLAB_TERRAFORM_STATE_STORAGE_PATH
}

gitlab_configure_lfs() {
if [[ ${GITLAB_LFS_OBJECT_STORE_ENABLED} == true ]]; then
echo "Configuring gitlab::lfs:object_store"
Expand Down Expand Up @@ -1503,11 +1544,11 @@ initialize_datadir() {
chown ${GITLAB_USER}: ${GITLAB_LFS_OBJECTS_DIR}

# create terraform_state directory
# TODO : parametarize path and replace with it (e.g. GITLAB_TERRAFORM_STATE_STORAGE_PATH) - see sameersbn/gitlab#2438
# TODO : wrap with "if [[ _ENABLED == true ]]" condition
mkdir -p ${GITLAB_SHARED_DIR}/terraform_state
chmod u+rwX ${GITLAB_SHARED_DIR}/terraform_state
chown ${GITLAB_USER}: ${GITLAB_SHARED_DIR}/terraform_state
if [[ ${GITLAB_TERRAFORM_STATE_ENABLED} == true ]]; then
mkdir -p ${GITLAB_TERRAFORM_STATE_STORAGE_PATH}
chmod u+rwX ${GITLAB_TERRAFORM_STATE_STORAGE_PATH}
chown ${GITLAB_USER}: ${GITLAB_TERRAFORM_STATE_STORAGE_PATH}
fi

# create registry dir
if [[ ${GITLAB_REGISTRY_ENABLED} == true ]]; then
Expand Down Expand Up @@ -1801,6 +1842,7 @@ configure_gitlab() {
gitlab_configure_ci
gitlab_configure_artifacts
gitlab_configure_packages
gitlab_configure_terraform_state
gitlab_configure_lfs
gitlab_configure_uploads
gitlab_configure_mattermost
Expand Down