[DCP Ingestion] Automate the DB initialization, Dockerize Ingestion Helper, and fix lock acquisition issue

[gmechali]

This PR does 3 main things:

• It dockerizes the ingestion helper such that we can run it in a Cloud Run Server. Today for base DC we deploy it as a Cloud Run Function, but that isnt ideal for DCP. For DCP, we will deploy a Cloud run server on the datacommon-ingestion-helper image. I have added a cloud build trigger in datcom-ci to rebuild the image everytime we make a change to the master branch on the ingestion-helper directory.
• Adds the Initialize Database action to the IngestionHelper function, which initializes all the tables in spanner_schema.sql (Edge, Node, Observation, ImportStatus, IngestionLock, ImportVersionHistory, IngestionHistory).
• Modifies the Lock Acquisition step to insert the lock row for the first time this is run.

As a follow up we should do the following:

• Migrate the Base DC deployed cloud run function to use this dockerized imaged and deploy it in cloud run server.
• Remove all the logic from the ingestion pipeline that initializes the database.

[gemini-code-assist]

Code Review

This pull request introduces a database initialization feature for the ingestion helper, including Docker and Cloud Build configurations, a Spanner schema, and logic to apply DDL statements. The locking mechanism was also updated to handle the creation of new lock rows. Feedback provides a suggestion for better DatabaseAdminClient configuration.

[dwnoble]

Thanks Gabe!

I think this tooling may be better suited to live in the DCP repo at https://github.com/datacommonsorg/datacommons

This PR converts the import automation cloud functions to a REST service, but if we're going to do that, I think it makes sense to consolidate it with the DCP FastAPI service so we don't end up adding a separate process.

Then this action could look something like: POST https://<dcp-service>/api/admin/initialize-database

Regardless, i think we should also:

• convert this toolset from requirements.txt files to uv so we can have consistent builds with the uv.lock files
• right now the functions in main.py are tied to cloud run functions. we should migrate these to a plain python module, and the cloud run part should be minimal wrapper on top of it
• we should expose these functions via a command line tool. If we migrate to the dcp repo, this might look like: $ datacommons admin initialize-database or $ datacommons admin release-ingestion-lock.

cc @vish-cs

[gmechali]

Thanks Gabe!

I think this tooling may be better suited to live in the DCP repo at https://github.com/datacommonsorg/datacommons

This PR converts the import automation cloud functions to a REST service, but if we're going to do that, I think it makes sense to consolidate it with the DCP FastAPI service so we don't end up adding a separate process.

Then this action could look something like: POST https://<dcp-service>/api/admin/initialize-database

Regardless, i think we should also:

• convert this toolset from requirements.txt files to uv so we can have consistent builds with the uv.lock files
• right now the functions in main.py are tied to cloud run functions. we should migrate these to a plain python module, and the cloud run part should be minimal wrapper on top of it
• we should expose these functions via a command line tool. If we migrate to the dcp repo, this might look like: $ datacommons admin initialize-database or $ datacommons admin release-ingestion-lock.

cc @vish-cs

Other comment kind of replies about moving to DCP repo. I think one day maybe but it's premature, and splitting up ingestion code also seems controversial.
Converting to UV - done!
Migrating from cloud functions to a plain python module, agreed but I dont think there's an immediate need for it? Can we track this post-M1?
Command line tool, again I think this is way past M1 right? All nice to haves, but are we aligned that these are future improvements, not for MVP?

[dwnoble]

Other comment kind of replies about moving to DCP repo. I think one day maybe but it's premature, and splitting up ingestion code also seems controversial.
Converting to UV - done!
Migrating from cloud functions to a plain python module, agreed but I dont think there's an immediate need for it? Can we track this post-M1?
Command line tool, again I think this is way past M1 right? All nice to haves, but are we aligned that these are future improvements, not for MVP?

Adding the python module & CLI later SGTM.

For converting to uv, we should also migrate from requirements.txt to pyproject.toml (example: https://github.com/datacommonsorg/datacommons/blob/main/packages/datacommons-db/pyproject.toml)

but that can also happen in a follow-up!

[gmechali]

Other comment kind of replies about moving to DCP repo. I think one day maybe but it's premature, and splitting up ingestion code also seems controversial.
Converting to UV - done!
Migrating from cloud functions to a plain python module, agreed but I dont think there's an immediate need for it? Can we track this post-M1?
Command line tool, again I think this is way past M1 right? All nice to haves, but are we aligned that these are future improvements, not for MVP?

Adding the python module & CLI later SGTM.

For converting to uv, we should also migrate from requirements.txt to pyproject.toml (example: https://github.com/datacommonsorg/datacommons/blob/main/packages/datacommons-db/pyproject.toml)

but that can also happen in a follow-up!

Changed to pyproject.toml!