18 Feb 18:18

github-actions

c8195b1

v0.2.16 Latest

Latest

Greenmask 0.2.16

Changes

Added Cloudflare R2 compatibility guide and fixed documentation typos #401
Fixed ACL entries restoration when using --restore-in-order #345
Re-structured contributor documentation #400
Enhanced and documented PostgreSQL version update procedure #383
Fixed deterministic order in PK/FK introspection query, resolving issues with composite foreign keys in the subset system #398
Updated sponsor logo #399

Full Changelog: v0.2.15...v0.2.16

Contributors

@jmmalmedy
@Dufran
@GreatBahram
@thomasbecker
@wwoytenko

Greenmask v1.0.0b1

Preface

This release marks the first beta version of Greenmask v1. It features a significant framework refactoring designed to enable support for multiple DBMSs by reusing core objects and abstractions.

This version introduces the MySQL engine. PostgreSQL support is not yet included in this release; the port of PostgreSQL support (based on v0.2.x) will be available soon.

Note: About 80% of the documentation for PostgreSQL is compatible with MySQL. We have not fully revised it for this release yet. If you have any questions or experience any issues, please open a GitHub issue or write to us on Discord.

Getting started

You can easily try out the MySQL engine using the provided Docker Compose playground.

Quick Start

Clone the repository and checkout the v1 branch:

git clone https://github.com/greenmaskio/greenmask.git
cd greenmask
git checkout v1

Run the playground environment:
```
docker compose -f v1/docker-compose-mysql.yml run greenmask-from-source
```
This command starts the source database, the transformation target database, and MinIO storage, and drops you into a shell inside the Greenmask container.

Run Greenmask commands:
Inside the container shell, you can run the following commands to dump, list, and restore data using the pre-mounted configuration:

# Create a dump
greenmask --config config.yaml dump

# List available dumps
greenmask --config config.yaml list-dumps

# Restore the latest dump
greenmask --config config.yaml restore latest

Playground Details

To run using the release image:

docker compose -f v1/docker-compose-mysql.yml run greenmask

The playground provides two database containers:

Original: Contains the source data.
Transformed: Intended target for restoration and verification.

Convenience aliases are available in the shell:

mysql_o: Connect to the original database.
mysql_t: Connect to the transformed database.
cleanup: Resets the transformed database (useful after restoring a dump to clean up for the next run).

The playground uses the Employees sample database for demonstration purposes.

You can view the full playground configuration file here.

Installation script

You can install greenmasl-mysql using install.sh script

curl -fsSL https://raw.githubusercontent.com/GreenmaskIO/greenmask/refs/heads/v1/install.sh | sh -s -- --binary greenmask-mysql --version v1.0.0b1

The binary will be installed as greenmask-mysql.

Docker images

You can use one of the image from dockerhub greenmask/greenmask:mysql-8.4-lts-v1.0.0b1 or greenmask/greenmask:mysql-8.0-v1.0.0b1 depending on which version is required for you.

Those images are also available in docker github registry mirror ghcr.io/greenmaskio/greenmask:mysql-8.4-lts-v1.0.0b1 or ghcr.io/greenmaskio/greenmask:mysql-8.0-v1.0.0b1.

Note: The Docker images are currently supported on linux/amd64 architecture only and are based on Debian Bookworm.

MySQL Implementation

Compatibility & Dependencies:

This release has been tested against MySQL 8.0 and 8.4. Other versions might work but have not been verified.

Requirements:

Greenmask focuses on data dumping and transformation, delegating schema operations to vendor tools. Ensure mysqldump (for dumping) and mysql client (for restoration) are installed and available in your environment.

Details:

Dump foramt: MySQL uses CSV format via LOAD DATA LOCAL INFILE statement. This approach is open for discussion; feel free to open a discussion on GitHub or Discord.

Status & Limitations

Some features are currently WIP for the MySQL engine:

show-dump command: Currently missing; intended to show details about a dump by its ID.
Template Language: Ported to support MySQL, but we are considering replacing it with an embedded JS interpreter for better predictability and user familiarity.
Integration Tests: Currently in progress.
Filtering: Including and excluding tables/schemas is currently limited. Regular expressions are not supported, and tables must be specified in the schema.table format.
Virtual References: Virtual references are not supported in this release.
Transformer Inheritance: Inheritance of transformers via Foreign Keys (FK/PK) is not supported.
Partitioned Tables: Inheritance of transformers for partitioned tables is not supported. Transformers must be defined for each partition if needed.
Global Salt: Setting a global salt variable for hash or deterministic engine transformers is currently not supported. This will be fixed in future versions.

Restore & Subset Limitations

Restoration Order: To restore data correctly and preserve Foreign Key integrity, you must use the --restore-in-order flag.
- This ensures tables are restored in topological order (for non-circular schemas).
- Performance Impact: This mode forces sequential processing (one worker), ignoring the --jobs flag.
- Circular Schemas: This mode may fail with circular schemas.
- Workaround: To speed up restoration, you can disable MySQL Foreign Key checks on the target database, allowing parallel restoration without this flag.
Circular References: The subset system currently cannot resolve circular schema dependencies and will panic in such cases. This is a known issue and will be fixed in future versions.

Documentation

The documentation has not yet been fully revised for v1/MySQL. However, 90% of the usage is identical to the PostgreSQL version.

Transformers: Transformers are identical. You can refer to the Transformers documentation for detailed usage.

Configuration

Greenmask v1 does not directly support standard MySQL environment variables. Instead, configuration is handled via the config file or environment variables using capitalized attribute names with dots replaced by underscores.

Common Options (dump.options): These parameters are common for all database engines.
MySQL Options (dump.mysql.options): These parameters are specific only to the MySQL engine.

Below is the list of supported settings for MySQL.

Environment Variables

Options can be set via environment variables by converting the YAML path to uppercase with underscores. For example, dump.mysql.options.user becomes DUMP_MYSQL_OPTIONS_USER.

Available Options

The options below correspond to standard mysqldump command arguments. However, only the options listed here are supported.

Dump Options (`dump.mysql.options`)

Connection: user, password, host, port, connect-database
General:
- all-databases: Dump all databases.
- databases: List of databases to dump.
- no-create-info: Exclude CREATE TABLE statements.
- no-data: Exclude data from dump.
- add-drop-table: Include DROP TABLE statements.
- compact: Reduce dump size with minimal comments.
- skip-comments: Do not include comments in dump.
- single-transaction: Use a single transaction for the dump.
- quick: Fetch rows one at a time.
- lock-tables: Lock all tables during dump.
Metadata:
- no-tablespaces: Exclude tablespace information.

Restore Options (`restore.mysql.options`)

Connection: user, password, host, port, connect-database
General:
- verbose: Enable verbose output.

Configuration Example

dump:
  tag:
    - "v1"
    - "daily"
  options:
    # Common Dump Options
    # Dump only data, without schema
    data-only: false                  # DUMP_OPTIONS_DATA_ONLY
    # Dump only schema, without data
    schema-only: false                # DUMP_OPTIONS_SCHEMA_ONLY
    # Number of parallel jobs
    jobs: 1                           # DUMP_OPTIONS_JOBS
    # Exclude table data (supported)
    exclude-table-data: []            # DUMP_OPTIONS_EXCLUDE_TABLE_DATA
    # Include/Exclude tables/schemas
    # Format: "schema.table" (Regexp is NOT supported)
    include-table:                    # DUMP_OPTIONS_INCLUDE_TABLE
      - "schema.table"
    exclude-table:                    # DUMP_OPTIONS_EXCLUDE_TABLE
      - "schema.table"
    include-schema:                   # DUMP_OPTIONS_INCLUDE_SCHEMA
      - "included_schema"
    exclude-schema:                   # DUMP_OPTIONS_EXCLUDE_SCHEMA
      - "excluded_schema"

  mysql:
    options:
      # Connection (Common to dump and restore)
      user: root                      # DUMP_MYSQL_OPTIONS_USER
      password: "1234"                # DUMP_MYSQL_OPTIONS_PASSWORD
      host: "playground-mysql-db-original" # DUMP_MYSQL_OPTIONS_HOST
      port: 3306                      # DUMP_MYSQL_OPTIONS_PORT
      connect-database: "employees"   # DUMP_MYSQL_OPTIONS_CONNECT_DATABASE

      # Dump Selection
      # Dump all databases (--all-databases)
      all-databases: false            # DUMP_MYSQL_OPTIONS_ALL_DATABASES
      # List of databases to dump
      databases:                      # DUMP_MYSQL_OPTIONS_DATABASES
        - ...

Contributors

wwoytenko and tarbaev-vl

Assets 13

0 Join discussion

13 Dec 15:46

github-actions

v0.2.15

ea5784e

v0.2.15

Greenmask 0.2.15

Changes

Supported PostgreSQL 18 #380
Added postcode type to Masking transformer for masking postcodes #362
Added skip_not_exist parameter to Json transformer operations. This allows skipping the operation if the key does not exist by the provided path #356
Fixed temp file cleanup order in schema-only dump #379
Fixed TOC entry duplication when merging schema and data sections #378

Full Changelog: v0.2.14...v0.2.15

Contributors

@bruno
@wwoytenko

Special thanks

@raayu83

Greenmask 0.2.14

Changes

Fixed RandomPerson hash generation #327
Implemented --quiet flag for list-dumps command #331. This makes it easy to use list-dumps in shell pipelines like:
```
greenmask list-dumps -q | xargs -I {} greenmask delete {}
```
Implemented an official greenmask installation script #334. Now greenmask can be installed with a single command:
```
curl -fsSL https://greenmask.io/install.sh | bash
```
Added a --description flag to the dump command, store it in metadata, and display it in list-dumps for better context #339.
Fixed logic in ExcludeSchema filter: now correctly returns false for excluded schemas, preventing them from being restored #343
Fix: ensure SEQUENCE SET and BLOB entries are restored after topologically sorted tables when using --restore-in-order #340
Fixed command links in index documentation #337
Fix: prevent panic when using latest dump id with restore command if no dumps exist in storage #346

Full Changelog: v0.2.13...v0.2.14

Contributors

@GreatBahram
@thejerrybao
@wwoytenko
@torgebauer
@atcruice

Greenmask 0.2.13

Changes

Fixed a panic in the introspection function when virtual references were set on tables without primary keys #309. Virtual references on such tables are still not supported, but the function no longer panics. Related MR #315.
Fixed a case when greenmask hash engine ignores GREENMASK_GLOBAL_SALT #317. Related MR #318.

Full Changelog: v0.2.12...v0.2.13

Contributors

@wwoytenko

Special thanks

@torgebauer
@k-ono65

Greenmask 0.2.12

Changes

Added support for dynamic parameters in the Replace transformer, allowing values to be dynamically replaced based on column values. This feature enables spreading the same value across multiple columns. See the documentation for examples. #293
Updated the --verbose flag to a boolean type. It is now true if provided and false otherwise. #282
Fixed a bug in the RandomDate transformer where minutes were not truncated as expected. #298
Updated go dependencies to the latest. #304

Full Changelog: v0.2.11...v0.2.12

Contributors

@mightydok
@wwoytenko

Special thanks

@edarchis
@mightydok
@floriandejonckheere

Greenmask 0.2.11

Changes

Catch up v0.2.10 lost changes. Change Oid type to uint32 #289.
Closes #284.

Full Changelog: v0.2.10...v0.2.11

Links

Feel free to reach out to us if you have any questions or need assistance:

Assets 13

05 Apr 11:58

github-actions

v0.2.10

dff586c

v0.2.10

Greenmask 0.2.10

Changes

Bump dependencies and upgraded go to 1.24 #285
Changed Oid type in TOC archive library #286. Closes #284
Revised documentation related to the log level values #287. Closes #283

Full Changelog: v0.2.9...v0.2.10

Contributors

@wwoytenko

Special thanks

@istvanp
@shkm

Greenmask 0.2.9

This release introduces a new transformer and fixes some bug.

Changes

Implemented RandomCompany transformer - it's a multi-column transformer, that generates a company data with attributes CompanyName and CompanySuffix
#273 #274.
Fixed a case when transformers with column containers were not printed on greenmask list-transformers command
call #275.
Fixed RandomEmail transformer bug when an incorrect buffer size for hex-encoded symbols resulted in a \0 appearing in the string #278.
Fixed typo in database_subset.md docs #271
Revised README.md #280

Full Changelog: v0.2.8...v0.2.9

Contributors

@edarchis
@mikfreedman
@wwoytenko

Special thanks

@edarchis
@mikfreedman

Greenmask 0.2.8

This release includes a new feature and bug fixes.

Changes

Support postgres:// schema connection URLs #263
Implemented --blobs and --no-blobs flags for the greenmask dump command. These flags allow inclusion or exclusion of large objects in the dump file. By default, --blobs is enabled. If --no-blobs is specified, large object data will be skipped, and only large object creation commands and ACLs will be included in the dump. #265
Implemented the --no-blobs flag for greenmask restore. If there are any large objects in your database, this will create an empty placeholder instead. #265
Enabled support for all textual data types in all transformers that have a text type #267. Closes #260
Fixed numerous introspection queries that have out of range errors caused by casting OID types to int4 values #264. Closes #265
Refactored the logic of restorers and added test coverage to improve maintainability and stability #268
Fixed domain constraint introspection query #266
Revised documentation according to the latest changes #267
Fixed brew build homebrew-core #206868 Closes #261

Uh oh!

Releases: GreenmaskIO/greenmask

v0.2.16

Greenmask 0.2.16

Changes

Full Changelog: v0.2.15...v0.2.16

Contributors

Links

Contributors

Uh oh!

v1.0.0b1

Greenmask v1.0.0b1

Preface

Getting started

Quick Start

Playground Details

Installation script

Docker images

MySQL Implementation

Compatibility & Dependencies:

Requirements:

Details:

Status & Limitations

Restore & Subset Limitations

Documentation

Configuration

Environment Variables

Available Options

Dump Options (dump.mysql.options)

Restore Options (restore.mysql.options)

Configuration Example

Contributors

Uh oh!

v0.2.15

Greenmask 0.2.15

Changes

Full Changelog: v0.2.14...v0.2.15

Contributors

Special thanks

Links

Contributors

Uh oh!

v0.2.14

Greenmask 0.2.14

Changes

Full Changelog: v0.2.13...v0.2.14

Contributors

Links

Contributors

Uh oh!

v0.2.13

Greenmask 0.2.13

Changes

Full Changelog: v0.2.12...v0.2.13

Contributors

Special thanks

Links

Contributors

Uh oh!

v0.2.12

Greenmask 0.2.12

Changes

Full Changelog: v0.2.11...v0.2.12

Contributors

Special thanks

Links

Contributors

Uh oh!

v0.2.11

Greenmask 0.2.11

Changes

Full Changelog: v0.2.10...v0.2.11

Links

Uh oh!

v0.2.10

Greenmask 0.2.10

Changes

Full Changelog: v0.2.9...v0.2.10

Contributors

Special thanks

Dump Options (`dump.mysql.options`)

Restore Options (`restore.mysql.options`)