At a glance
ProblemAgent pod is killed and restarted under concurrent-session load; separately, the /budget endpoint fails.
AffectsEnterprise h2oGPTe 1.6.x on Kubernetes
WorkaroundYes — probe tuning, and a settings change for /budget
SolutionProbe tuning (permanent); upgrade to 1.7.x for the /budget defect

Problem

An Enterprise h2oGPTe agent pod is terminated unexpectedly under concurrent-session load. Container logs show only a SIGTERM entry, with no preceding error:

[<timestamp>] INFO - HTTP Request: GET http://<file-service>:8100/v1/files "HTTP/1.1 200 OK"
[<timestamp>] [INFO] Handling signal: term

kubectl describe pod may initially show Events: <none>, but on later inspection reveals:

Warning  Unhealthy  Liveness probe failed:  Get <URL>: context deadline exceeded
Warning  Unhealthy  Readiness probe failed: Get <URL>: context deadline exceeded
Normal   Killing    kubelet spec.containers{agent}: Container agent failed liveness probe, will be restarted

A second, unrelated symptom may appear in the same environment: the /budget endpoint fails while other file-service requests succeed.

Affected versions

ProductEnterprise h2oGPTe 1.6.x (observed on 1.6.58)
ComponentAgent shared pod
DeploymentKubernetes
TriggerSustained concurrent user sessions against a small number of agent replicas
Fixed in1.7.x — for the /budget defect only. Probe tuning applies to all versions.

Issue 1 — Pod killed by liveness probe

Cause

The kubelet killed the agent container after its liveness probe timed out. The default probe timeout of 2 seconds is too aggressive for the agent pod when many sessions run concurrently: a busy but perfectly healthy agent cannot answer /health within it, so Kubernetes treats it as failed and restarts it.

Default configuration, for reference:

Liveness:  http-get http://:5004/health  delay=10s timeout=2s period=10s #success=1 #failure=5
Readiness: http-get http://:5004/health  delay=10s timeout=2s period=10s #success=2 #failure=5
Startup:   http-get http://:5004/health  delay=30s timeout=5s period=10s #success=1 #failure=180
Solution — tune the probes

Increase the probe tolerance for the agent container in your Helm values. Three settings matter, in this order:

  • timeoutSeconds — the primary fix. Raise it until it comfortably exceeds the slowest /health response you observe at peak load.
  • periodSeconds. Raise alongside the timeout so probes are not issued faster than the pod can answer. Keep it at or above timeoutSeconds, or probes queue against an already-saturated pod and worsen the problem.
  • failureThreshold. Leave enough tolerance that one transient slow response does not trigger a restart.

Size these against your own environment rather than copying fixed numbers. Measure how long /health takes to respond during your peak concurrency, then set timeoutSeconds with clear headroom above that figure. Apply the same treatment to both probes.

livenessProbe:
  type: httpGet
  path: /health
  port: 5004
  timeoutSeconds: <well above peak /health response time>
  periodSeconds: <at or above timeoutSeconds>
  failureThreshold: <tolerance for transient spikes>

readinessProbe:
  type: httpGet
  path: /health
  port: 5004
  timeoutSeconds: <well above peak /health response time>
  failureThreshold: <tolerance for transient spikes>
If probe failures continue

The pod is genuinely saturated rather than being killed prematurely. Scale out by increasing the replica count for the agent shared deployment. Probe tuning stops healthy pods being killed — it does not add capacity.

Issue 2 — /budget endpoint fails

Cause

A defect in the 1.6.x line causes /budget to fail when Max LLM Cost Per User is set to a negative value (-1, meaning unlimited).

Workaround

In the admin UI under Settings, set Max LLM Cost Per User to a large positive number (for example 9999999) instead of -1.

Solution

Upgrade to Enterprise h2oGPTe 1.7.x, where handling of unlimited budgets was standardised. See the 1.7 changelog.

How to verify

  1. Confirm the new probe values are live on the container: 
    kubectl describe pod <pod-name> -n <namespace> | grep -A3 -i "liveness\|readiness"
  2. Monitor restart counts across a representative load period — they should stop incrementing: 
    kubectl get pods -n <namespace> -w
  3. Confirm no new Unhealthy or Killing events are recorded: 
    kubectl get events -n <namespace> --field-selector reason=Unhealthy
  4. For the budget issue, load the Settings page and confirm /budget returns successfully.

Related