Lock remains in HMS if HiveTableOperations gets killed (direct process shutdown - no signals) after lock is acquired

[pPanda-beta]

Although unlock is kept inside the finally block,

iceberg/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java

Line 220 in a790199

cleanupMetadataAndUnlock(threw, newMetadataLocation, lockId);

YET INFINITE LOCK MAY HAPPEN

Steps to reproduce :

1. Start a new commit
2. Let icberg acquire the lock
3. Kill the job (signal 9 / 19 ) or disconnect network. (See below to know more about actual scenarios that happened)
4. Restart the job, a fresh new commit
5. Thats it, it will never be able to acquire a new lock on same resource again

Lets make it more spicy

Consider it as the table create operation 🥰 , i.e. the actual "table" does not exist yet and before creation it will be killed.

Steps:

1. Start creating the table
2. Let it acquire the lock on the non-existent table resource (HMS allows that)
3. Kill the job before the table is even created
4. Go cry yourself, cause you can not remove this lock manually using beeline / hive-cli since table doesn’t exist.

What is this "Killed" thingy?

Well here in rapido we use GCP preemptible VMs for low cost. Now on such kind of infra, VM may be preempted at any point of time with very short notice period (15 sec).
Consider this as a network cable unplug. We will never get a chance for InterruptedException and do clean up operations.

Why hive never faced this issue?

Well hive has expiry of its locks which works as a TTL. So system recovers eventually.

Disaster Recoveries that we tried so far:

1. Ensure no jobs are running for that table (which may not have been created yet)
2. Use java api to connect to the hive metastore
3. Delete all locks for that table

Suggestions

1. Before acquiring locks, delete all locks that have not received any heartbeats since last 'x' minutes. (configurable)
2. After acquiring lock keep sending heart beats to HMS from a different daemon thread. This will ensure concurrent writes are protected
3. After operation finishes (success or fail), unlock and cancel the heartbeat thread.

[pvary]

Quick question: Hive removes the locks by the AcidHouseKeeperService which should run on the HMS side. Shouldn't that remove the "forever" from the problem statement?

The commits should be fairly quick so having a small timeout could be acceptable if HMS is only used as a HiveCatalog for Iceberg.

[marton-bod]

What's the value of metastore.txn.timeout on your HMS instance? As @pvary suggested, the AcidHouseKeeperService should clean up any locks that haven't heartbeated within the timeout period (which is 300 seconds by default), preventing the forever-blockage. The housekeeper service itself will run every 180 seconds by default to purge these timed-out locks (configured by metastore.timedout.txn.reaper.interval on Hive 3.1). Maybe it's worth checking the HMS logs also to double check if the house keeper service is running regularly

[pPanda-beta]

@pvary @marton-bod
Thanks for the suggestion.
I've gone through their implementation.

This does cleanup via:

1. https://github.com/apache/hive/blob/8190d2be7b7165effa62bd21b7d60ef81fb0e4af/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/AcidHouseKeeperService.java#L60

2. https://github.com/apache/hive/blob/8190d2be7b7165effa62bd21b7d60ef81fb0e4af/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L4403

and

3. https://github.com/apache/hive/blob/8190d2be7b7165effa62bd21b7d60ef81fb0e4af/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L3633-L3636

But I believe ALL OF THESE WONT WORK:
cause iceberg does not create transactions before creating the locks.

iceberg/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java

Lines 322 to 329 in a790199

	final LockComponent lockComponent = new LockComponent(LockType.EXCLUSIVE, LockLevel.TABLE, database);
	lockComponent.setTablename(tableName);
	final LockRequest lockRequest = new LockRequest(Lists.newArrayList(lockComponent),
	System.getProperty("user.name"),
	InetAddress.getLocalHost().getHostName());
	LockResponse lockResponse = metaClients.run(client -> client.lock(lockRequest));
	AtomicReference<LockState> state = new AtomicReference<>(lockResponse.getState());
	long lockId = lockResponse.getLockid();

This is more like a transaction less lock 😭 . So the lifecycle needs to be maintained via iceberg only.

Also I would insist approaches which are heavily involving hive metastore features only, acid has mostly came from a hive feature. In modern days no one even installs hive. People mostly use spark/presto on top of HMS.

[pPanda-beta]

While dealing with independent locks we have to implement with the following columns only,

| lockid | database | table | partition | lock_state | blocked_by | lock_type | transaction_id | last_heartbeat | acquired_at | user | hostname | agent_info

[marton-bod]

Hi @pPanda-beta
Transaction-less locks (where transaction id is 0) should be cleaned up the same way, once the timeout period expires.
They are cleaned at this point: https://github.com/apache/hive/blob/8190d2be7b7165effa62bd21b7d60ef81fb0e4af/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L4400
even one step before the timed-out transactions are cleaned up.
See: https://github.com/apache/hive/blob/8190d2be7b7165effa62bd21b7d60ef81fb0e4af/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L4316

[pPanda-beta]

@marton-bod Thanks for pointing it out.
Theoretically this should work.

I've the value of metastore.task.threads.remote as org.apache.hadoop.hive.metastore.txn.AcidHouseKeeperService,org.apache.hadoop.hive.metastore.txn.AcidOpenTxnsCounterService,org.apache.hadoop.hive.metastore.txn.AcidCompactionHistoryService,org.apache.hadoop.hive.metastore.txn.AcidWriteSetService,org.apache.hadoop.hive.metastore.MaterializationsRebuildLockCleanerTask

And I dont see the logs "Failed to purge timedout locks due to".
Definitely something is broken on my HMS.

Let me investigate that. Meanwhile we can remove the "forever" part.

[pPanda-beta]

@marton-bod Found it in an integration test powered by embed hive metastore.
We need
metastore.compactor.initiator.on=true
At least on embedded HMS, this is false by default, which was not running those threads.

[marton-bod]

You're correct, it needs to be turned on for the house keeper to be launched.

[markope]

Is there any workaround for this? How could I manually fix my HMS?

[szehon-ho]

I found it useful to have a small script to unlock manually as well (in case the HMS thread is not configured). In scala for instance assuming you get a hmsclient:

 def unlock(db: String, table: String): Unit = {
    val lockIds = getLockIds(db, table)
    lockIds.foreach{
      hmsClient.unlock(_)
    }
  }

  private def getLockIds(db: String, table: String): Set[Long] = {
    val showLockRequest = new ShowLocksRequest()
    showLockRequest.setDbname(db)
    showLockRequest.setTablename(table)
    showLockRequest.setIsExtended(true)
    val response = hmsClient.showLocks(showLockRequest)
    val locks = response.getLocks.asScala
    locks.map(_.getLockid).toSet
  }

[lirui-apache]

We also encountered this issue with iceberg 0.11.1. Intuitively I wanted to enable the house keeper service in HMS. But then I noted iceberg doesn't send heartbeat for acquired locks, which seems to imply it relies on HMS not running the house keeper services. Wondering whether that's by design.

[pvary]

Heartbeat is usually not needed for Iceberg table locks. The lock is only kept for the atomicity of the snapshot swap. This should be almost immediate. Heathbeat interval is 5 min, which means no heartbeat is sent with the default conf anyway.

[lirui-apache]

Thanks @pvary for the clarifications. Then I'll go with the house keeper service.