Import/export support for emulators

[samtstern]

Description

(Fixes #1167)

TODO:

• API review confirmation
• Changelog

This pull request imports import/export, currently only supporting the Firestore emulator but doing so in a general way.

Emulator Hub

This PR introduces the "emulator hub" which is a new local service that has the same behavior patterns as any other emulator but does not emulate a real Firebase service. Instead it is a local REST API which can be used for controlling running emulators.

When the hub starts it writes a file to /tmp/hub-$PROJECTID.json that looks like this:

{"version":"7.12.1","host":"localhost","port":4000}

This allows other instances of the CLI to discover where the hub is running for a given project. As of this PR the hub exposes two endpoints:

GET /
Used for a hub health check. Returns exactly what is in the tmp json file above.

POST /_admin/export
Kick off a data export of all running emulators. Takes a JSON body with a single path field.

These endpoints are not considered a public API at this time, although they may be one in the future.

Import data

This adds a new --import flag to both emulators:start and emulators:exec

Export data

Adds a new command firebase emulators:export which discovers the emulator hub of another running emulators process and hits the admin export endpoint.

Scenarios Tested

Start with import

$ firebase emulators:start --import=./data --only firestore
i  emulators: Starting emulators: firestore
✔  emulators: Emulator hub started at http://localhost:4000
i  firestore: Importing data from /tmp/tmp.bqjxD5UtNv/data/firestore_export/firestore_export.overall_export_metadata
//...
✔  firestore: Emulator started at http://localhost:8080
i  firestore: For testing set FIRESTORE_EMULATOR_HOST=localhost:8080
✔  All emulators started, it is now safe to connect.

Export (clean)

$ firebase  emulators:export ./data/
i  Found running emulator hub for project fir-dumpster at http://localhost:4000
i  Creating export directory /tmp/tmp.bqjxD5UtNv/data
i  Exporting data to: /tmp/tmp.bqjxD5UtNv/data
✔  Export complete

$ cat ./data/firebase-export-metadata.json  | jq
{
  "version": "7.13.0",
  "firestore": {
    "version": "1.10.4",
    "path": "firestore_export",
    "metadata_file": "firestore_export/firestore_export.overall_export_metadata"
  }
}

Export (overwrite prompt)

$ firebase  emulators:export ./data/
i  Found running emulator hub for project fir-dumpster at http://localhost:4000
? The directory /tmp/tmp.bqjxD5UtNv/data already contains export data. Exporting again to the same directory w
ill overwrite all data. Do you want to continue? Yes
i  Deleting directory /tmp/tmp.bqjxD5UtNv/data/firestore_export
i  Exporting data to: /tmp/tmp.bqjxD5UtNv/data
✔  Export complete

Export (--force)

$ firebase  emulators:export --force ./data/
i  Found running emulator hub for project fir-dumpster at http://localhost:4000
i  Deleting directory /tmp/tmp.bqjxD5UtNv/data/firestore_export
i  Exporting data to: /tmp/tmp.bqjxD5UtNv/data
✔  Export complete

Sample Commands

See above (scenarios tested)

[samtstern]

@yuchenshi @abeisgoat apologies this PR got a little larger than I would have liked because it involved implementing the hub and also some new end-to-end testing logic. The core is not too complex though.

[yuchenshi]

Good work!

[yuchenshi]

Purrrrfect.

[yuchenshi]

Integration test?

[dazziola]

PLEASE IGNORE - EDIT - Turns out I had moved my partial exports from their original root folder. All works now, thanks for working on this really useful feature @samtstern !

I've exported my "production" Firestore data using gcloud firestore export gs://... and have received my Firestore in output-0 -> output-7 files, with a top-level metadata file.

I've added this to my firebase project in a seed folder, and have added a firebase-export-metadata.jsonfile to the same seed folder with contents:

{
    "version": "7.14.0",
    "firestore": {
        "version": "1.10.4",
        "path": "",
        "metadata_file": "2020-03-01T20_00_54_25297_all_namespaces_all_kinds_all_namespaces_all_kinds.export_metadata"
    }
}

When I attempt to start only the Firestore emulator (firebase emulators:start --import=seed --only firestore), I get this in my firestore-debug.log file:

Exception in thread "main" com.google.cloud.datastore.core.exception.DatastoreException: Invalid record
	at com.google.cloud.datastore.emulator.impl.LevelDBLogReaderChannel.readPhysicalRecord(LevelDBLogReaderChannel.java:217)
	at com.google.cloud.datastore.emulator.impl.LevelDBLogReaderChannel.readRecord(LevelDBLogReaderChannel.java:88)
	at com.google.cloud.datastore.emulator.impl.LevelDBLogReaderChannel.readString(LevelDBLogReaderChannel.java:70)
	at com.google.cloud.datastore.emulator.impl.ExportImportUtil.parseOverallMetadataFile(ExportImportUtil.java:218)
	at com.google.cloud.datastore.emulator.impl.ExportImportUtil.fetchEntities(ExportImportUtil.java:54)
	at com.google.cloud.datastore.emulator.firestore.CloudFirestore.main(CloudFirestore.java:89)

Any ideas? My Firestore export has not been changed from what was exported thru gcloud CLI.

[samtstern]

@dazziola could you please file a new issue for this? Also we'll need the following extra info:

• The whole --debug log output of your firebase emulators:start --import=seed command
• The output of ls -alR seed

Thanks!