Received message larger than max (4237525 vs. 4194304) - Python client cannot list experiments

[CharlesPDX]

After recording ~130 experiments, attempting to list them from the Python client fails with an oversized message exception. Adding a filter does not work around the issue.

Traceback when calling keepsake.experiments.list():

_InactiveRpcError Traceback (most recent call last)
~/auto-stop-tar/.venv/lib/python3.9/site-packages/keepsake/daemon.py in wrapped(*args, **kwargs)
31 try:
---> 32 return f(*args, **kwargs)
33 except grpc.RpcError as e:

~/auto-stop-tar/.venv/lib/python3.9/site-packages/keepsake/daemon.py in list_experiments(self)
208 def list_experiments(self) -> List[Experiment]:
--> 209 ret = self.stub.ListExperiments(pb.ListExperimentsRequest())
210 return pb_convert.experiments_from_pb(self.project, ret.experiments)

~/auto-stop-tar/.venv/lib/python3.9/site-packages/grpc/_channel.py in call(self, request, timeout, metadata, credentials, wait_for_ready, compression)
945 wait_for_ready, compression)
--> 946 return _end_unary_response_blocking(state, call, False, None)
947

~/auto-stop-tar/.venv/lib/python3.9/site-packages/grpc/_channel.py in _end_unary_response_blocking(state, call, with_call, deadline)
848 else:
--> 849 raise _InactiveRpcError(state)
850

_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.RESOURCE_EXHAUSTED
details = "Received message larger than max (4237525 vs. 4194304)"
debug_error_string = "{"created":"@1645296119.443456300","description":"Received message larger than max (4237525 vs. 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":205,"grpc_status":8}"

During handling of the above exception, another exception occurred:

Exception Traceback (most recent call last)
/tmp/ipykernel_4297/1455666872.py in
----> 1 experiments = keepsake.experiments.list(lambda e: "autotar" in e.params["run_group"])
2 len(experiments)

~/auto-stop-tar/.venv/lib/python3.9/site-packages/keepsake/experiment.py in list(self, filter)
387 Return all experiments for a project, sorted by creation date.
388 """
--> 389 experiments = self.project._daemon().list_experiments()
390 result: ExperimentList = ExperimentList()
391 for exp in experiments:

~/auto-stop-tar/.venv/lib/python3.9/site-packages/keepsake/daemon.py in wrapped(*args, **kwargs)
38 if status_code:
39 raise handle_exception(status_code, details)
---> 40 raise Exception(details)
41
42 return wrapped

Exception: Received message larger than max (4237525 vs. 4194304)

[CharlesPDX]

For anyone else encountering this, the work-around change that's in the PR can be done locally (at your own risk). Eventually it probably starts to run into the 15 second timeout, so that would need to be bumped/configurable. A longer-term solution might be implementing a paging protocol, or filtering that can be passed to the daemon like the CLI (but with more robust filtering).

[zeke]

Hi @CharlesPDX 👋🏼

The Keepsake project is no longer actively maintained. If you're interested in helping maintain it, please let us know.