terraform-aws-cloudfront-access-logs-to-hive-format

A Terraform template that move CloudFront Access Logs to Hive format for Athena.

Reference

• aws-samples/amazon-cloudfront-access-logs-queries
  • "This moves the file to an Apache Hive style prefix." part

Changes

• CloudFormation stack to TerraForm
• Use only moveAccessLogs
• Keep the prefix and put it in front of the datetime

Overview

• e.g.
  • CloudFront logging setting:
    • S3 bucket: example-bucket
    • Log prefix: raw/example.com
  • Inputs:
    • new_key_prefix: raw/
    • gz_key_prefix: partitioned-gz/
  • Result:
    • source:
      • s3://example-bucket/raw/example.com/XXXXXXXXXXXXX.1990-01-01-00.XXXXXXXX.gz
    • destination:
      • S3: s3://example-bucket/partitioned-gz/example.com/year=1990/month=01/day=01/hour=00/XXXXXXXXXXXXX.1990-01-01-00.XXXXXXXX.gz

Requirements

Name	Version
terraform	>= 1.0.4
archive	>= 2.2
aws	>= 3.52

Providers

Name	Version
archive	>= 2.2
aws	>= 3.52

Modules

No modules.

Resources

Name	Type
aws_cloudwatch_log_group.this	resource
aws_iam_role.this	resource
aws_iam_role_policy.this	resource
aws_iam_role_policy_attachment.lambda_basic_execution_role	resource
aws_kms_alias.this	resource
aws_kms_key.this	resource
aws_lambda_function.this	resource
aws_lambda_permission.this	resource
aws_s3_bucket_notification.this	resource
archive_file.this	data source
aws_caller_identity.current	data source
aws_iam_policy_document.this	data source
aws_iam_policy_document.this_assume_role_policy	data source
aws_iam_policy_document.this_kms_key_policy	data source
aws_region.current	data source

Inputs

Name	Description	Type	Default	Required
create_s3_bucket_notification	S3 Buckets only support a single notification configuration. ref.aws_s3_bucket_notification	bool	false	no
lambda_function_log_retention_in_days	Possible values are: 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 3653, and 0(never)	number	30	no
lambda_function_runtime	Identifier of the function's runtime.	string	"nodejs16.x"	no
lambda_function_timeout	Amount of time your Lambda Function has to run in seconds.	number	30	no
name_prefix	Prefix that is used for the created resources. If specified, it will be '{name_prefix}-{name_suffix}'	string	""	no
name_suffix	Suffix that is used for the created resources.	string	"move-cf-access-logs"	no
s3_bucket_gz_key_prefix	Prefix of gzip'ed access log files that are moved to the Apache Hive like style. Including the trailing slash	string	"partitioned-gz/"	no
s3_bucket_name	Bucket Name of access log files that are written by Amazon CloudFront	string	n/a	yes
s3_bucket_new_key_prefix	Prefix of new access log files that are written by Amazon CloudFront. Including the trailing slash.	string	"raw/"	no

Outputs

Name	Description
function_arn	The ARN of the Lambda Function