Skip to content

feat: Adding new dataflow pipelines to import HBase snapshot and import validation #2958

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vermas2012 merged 2 commits into from
May 11, 2021
Merged

feat: Adding new dataflow pipelines to import HBase snapshot and import validation #2958

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d582cf2
feat: Ability to import HBase Snapshot data into Cloud Bigtable using…
vermas2012 Feb 1, 2021
16a0731
feat: Add a new pipeline to validate data imported from HBase (#2828)
vermas2012 Feb 12, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions .kokoro/nightly/integration-beam.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Format: //devtools/kokoro/config/proto/build.proto
Comment thread
vermas2012 marked this conversation as resolved.

# Configure the docker image for kokoro-trampoline.
env_vars: {
key: "TRAMPOLINE_IMAGE"
value: "gcr.io/cloud-devrel-kokoro-resources/java8"
}

env_vars: {
key: "INTEGRATION_TEST_ARGS"
value: "-PbeamIntegrationTest -Dgoogle.bigtable.project.id=gcloud-devel -Dgoogle.bigtable.instance.id=google-cloud-bigtable -Dgoogle.dataflow.stagingLocation=gs://java-bigtable-hbase-testing/staging -Dcloud.test.data.folder=gs://java-bigtable-hbase-testing/hbase-snapshot-import-integration-tests -Dregion=us-central1"
}

env_vars: {
key: "JOB_TYPE"
value: "integration"
}

# TODO: remove this after we've migrated all tests and scripts
env_vars: {
key: "GCLOUD_PROJECT"
value: "gcloud-devel"
}

env_vars: {
key: "GOOGLE_CLOUD_PROJECT"
value: "gcloud-devel"
}

env_vars: {
key: "GOOGLE_APPLICATION_CREDENTIALS"
value: "secret_manager/java-it-service-account"
}

env_vars: {
key: "SECRET_MANAGER_KEYS"
value: "java-it-service-account"
}
38 changes: 38 additions & 0 deletions .kokoro/presubmit/integration-beam.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Format: //devtools/kokoro/config/proto/build.proto

# Configure the docker image for kokoro-trampoline.
env_vars: {
key: "TRAMPOLINE_IMAGE"
value: "gcr.io/cloud-devrel-kokoro-resources/java8"
}

env_vars: {
key: "INTEGRATION_TEST_ARGS"
value: "-PbeamIntegrationTest -Dgoogle.bigtable.project.id=gcloud-devel -Dgoogle.bigtable.instance.id=google-cloud-bigtable -Dgoogle.dataflow.stagingLocation=gs://java-bigtable-hbase-testing/staging -Dcloud.test.data.folder=gs://java-bigtable-hbase-testing/hbase-snapshot-import-integration-tests -Dregion=us-central1"
}

env_vars: {
key: "JOB_TYPE"
value: "integration"
}

# TODO: remove this after we've migrated all tests and scripts
env_vars: {
key: "GCLOUD_PROJECT"
value: "gcloud-devel"
}

env_vars: {
key: "GOOGLE_CLOUD_PROJECT"
value: "gcloud-devel"
}

env_vars: {
key: "GOOGLE_APPLICATION_CREDENTIALS"
value: "secret_manager/java-it-service-account"
}

env_vars: {
key: "SECRET_MANAGER_KEYS"
value: "java-it-service-account"
}
9 changes: 5 additions & 4 deletions bigtable-dataflow-parent/bigtable-beam-import/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ This project supports importing and exporting HBase Sequence Files to Google Clo
Cloud Dataflow.

## Instructions

Download [the import/export jar](http://search.maven.org/remotecontent?filepath=com/google/cloud/bigtable/bigtable-beam-import/1.1.0/bigtable-beam-import-1.1.0-shaded.jar), which is an aggregation of all required jars.
[//]: # ({x-version-update-start:bigtable-dataflow-parent:released})
Download [the import/export jar](http://search.maven.org/remotecontent?filepath=com/google/cloud/bigtable/bigtable-beam-import/1.19.3/bigtable-beam-import-1.19.3-shaded.jar), which is an aggregation of all required jars.

Please pay attention to the Cluster CPU usage and adjust the number of Dataflow workers accordingly.

Expand All @@ -14,7 +14,7 @@ Please pay attention to the Cluster CPU usage and adjust the number of Dataflow
On the command line:

```
java -jar bigtable-beam-import-1.1.0-shaded.jar export \
java -jar bigtable-beam-import-1.19.3-shaded.jar export \
--runner=dataflow \
--project=[your_project_id] \
--bigtableInstanceId=[your_instance_id] \
Expand All @@ -32,7 +32,7 @@ Create the table in your cluster.
On the command line:

```
java -jar bigtable-beam-import-1.1.0-shaded.jar import \
java -jar bigtable-beam-import-1.19.3-shaded.jar import \
--runner=dataflow \
--project=[your_project_id] \
--bigtableInstanceId=[your_instance_id] \
Expand All @@ -42,3 +42,4 @@ java -jar bigtable-beam-import-1.1.0-shaded.jar import \
--maxNumWorkers=[3x number of nodes] \
--zone=[zone of your cluster]
```
[//]: # ({x-version-update-end})
115 changes: 108 additions & 7 deletions bigtable-dataflow-parent/bigtable-beam-import/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ limitations under the License.
<artifactId>bigtable-beam-import</artifactId>

<properties>
<mainClass>com.google.cloud.bigtable.beam.sequencefiles.Main</mainClass>
<mainClass>com.google.cloud.bigtable.beam.Main</mainClass>
</properties>

<!-- Adding this to resolve version conflict within beam sdk-->
Expand All @@ -46,6 +46,12 @@ limitations under the License.
<groupId>${project.groupId}</groupId>
<artifactId>bigtable-hbase-beam</artifactId>
<version>${project.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-shaded-client</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.google.cloud.bigtable</groupId>
Expand All @@ -64,9 +70,12 @@ limitations under the License.
<groupId>io.opencensus</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-shaded-client</artifactId>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-core</artifactId>
Expand All @@ -84,10 +93,18 @@ limitations under the License.
<artifactId>beam-sdks-java-io-hadoop-common</artifactId>
<version>${beam.version}</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-hadoop-format</artifactId>
<version>${beam.version}</version>
</dependency>

<!-- For HBase 2.x, this should be hbase-mapreduce
https://hbase.apache.org/2.1/book.html#export
-->
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-shaded-client</artifactId>
<artifactId>hbase-shaded-server</artifactId>
<version>${hbase.version}</version>
</dependency>

Expand All @@ -101,7 +118,7 @@ limitations under the License.
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>${beam-guava.version}</version>
<version>${gcs-guava.version}</version>
</dependency>

<!-- TODO: check if commons-codec was transitively updated to 1.13 and okhttp was updated to 2.7.5 when upgrading-->
Expand All @@ -121,6 +138,12 @@ limitations under the License.
</exclusions>
</dependency>

<dependency>
<groupId>com.google.code.findbugs</groupId>
<artifactId>jsr305</artifactId>
<version>${jsr305.version}</version>
</dependency>

<!-- TODO: remove these dependencies when upgraded through transitive dependency (beam-runners-google-cloud-dataflow-java)
these are not used directly, but upgrading due to transitive vulnerabilities in older versions-->
<dependency>
Expand All @@ -146,6 +169,21 @@ limitations under the License.
<artifactId>slf4j-api</artifactId>
<version>${slf4j.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/gcs-connector -->
<dependency>
<groupId>com.google.cloud.bigdataoss</groupId>
<artifactId>gcs-connector</artifactId>
<version>hadoop2-2.1.4</version>
<classifier>shaded</classifier>
</dependency>

<!-- https://mvnrepository.com/artifact/com.google.apis/google-api-services-storage -->
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-storage</artifactId>
<version>v1-rev171-1.25.0</version>
</dependency>


<!-- Test -->
<dependency>
Expand Down Expand Up @@ -178,6 +216,24 @@ limitations under the License.
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-shaded-testing-util</artifactId>
<version>${hbase.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.google.truth</groupId>
<artifactId>truth</artifactId>
<version>1.0.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-bigtable-emulator</artifactId>
<version>0.124.0</version>
<scope>test</scope>
</dependency>
</dependencies>

<build>
Expand Down Expand Up @@ -213,7 +269,7 @@ limitations under the License.
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.6.0</version>
<version>3.0.0</version>
<executions>
<execution>
<goals>
Expand All @@ -229,7 +285,7 @@ limitations under the License.
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>3.0.0</version>
<version>3.2.0</version>
<executions>
<execution>
<id>add-source</id>
Expand Down Expand Up @@ -262,6 +318,16 @@ limitations under the License.
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
</transformers>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</plugin>

Expand Down Expand Up @@ -332,7 +398,7 @@ limitations under the License.
<configuration>
<forkCount>1</forkCount>
<includes>
<include>**/*IT.java</include>
<include>**/sequencefiles/*IT.java</include>
</includes>
<!-- Use Isolated Classloader so that dataflow can find all files
that must be staged.
Expand All @@ -346,6 +412,7 @@ limitations under the License.
</build>
</profile>

<!-- profile to run all integration tests -->
<profile>
<id>beamIntegrationTest</id>
<build>
Expand All @@ -364,6 +431,7 @@ limitations under the License.
<forkCount>1</forkCount>
<includes>
<include>**/CloudBigtableBeamITTest.java</include>
<include>**/*IT.java</include>
</includes>
<reportNameSuffix>bigtable-beam</reportNameSuffix>
</configuration>
Expand All @@ -373,5 +441,38 @@ limitations under the License.
</plugins>
</build>
</profile>

<profile>
<id>hbasesnapshotsIntegrationTest</id>
<properties>
<!-- Needed to enable integration tests before we figure out -->
<!-- possible misconfiguration of Failsafe plugin -->
<skipITs>false</skipITs>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<id>hbasesnapshots-integration-test</id>
<goals>
<goal>integration-test</goal>
</goals>
<phase>integration-test</phase>
<configuration>
<forkCount>1</forkCount>
<includes>
<include>**/hbasesnapshots/*IT.java</include>
</includes>
<useSystemClassLoader>false</useSystemClassLoader>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>
Loading