Machine intelligence for your business

Problem

The maximum collection size is increased on the platform — for example from 1 GB to 10 GB — usually as part of an upgrade or a configuration change.

Collections created after the change show the new limit. Collections created before it still show and enforce the previous limit, and users working in them keep hitting the old ceiling.

There is no error message to search for. The symptom is an upload that stops at the old limit in a collection whose settings still read the old value, on a platform where the limit has already been raised.

On a deployment with many pre-existing collections this affects every one of them, and the UI offers no way to correct them in bulk.

Affected versions

Cause

A collection's size limit is a property of that collection, recorded when the collection is created. The platform-level default supplies the starting value for newly created collections only — it is not re-evaluated against collections that already exist.

Raising it therefore changes the default for future collections and leaves existing ones untouched. Each existing collection has to be updated individually.

Workaround

Solution

Before you start — API key handling

• Create a dedicated global API key for this task rather than reusing an existing one.
• Pass it through an environment variable, not by editing it into the script. A key pasted into a file tends to survive in shell history, editor backups and version control.
• Never print or log the key, and do not paste it into chat, tickets or screenshots.
• Revoke the key as soon as the work is finished — delete it from the APIs page in the account menu.

Script

Set the mode at the top and run each stage in order. The script reads the address and key from the environment:

export H2OGPTE_ADDRESS="https://your-h2ogpte-host"
export H2OGPTE_API_KEY="..."     # dedicated key, revoke when done

import os
from h2ogpte import H2OGPTE

MODE = "dry-run"          # "dry-run" -> "pilot" -> "apply"

GB = 1000 ** 3            # decimal GB, matches what h2oGPTe displays
THRESHOLD = 10 * GB       # update collections below this
NEW_LIMIT = 10 * GB       # set them to this

client = H2OGPTE(
    address=os.environ["H2OGPTE_ADDRESS"],
    api_key=os.environ["H2OGPTE_API_KEY"],
)


def fmt(n):
    if n is None:
        return "unlimited"
    v = float(n)
    for unit in ("B", "KB", "MB", "GB", "TB"):
        if v < 1000:
            return f"{v:g} {unit}"
        v /= 1000
    return f"{v:g} PB"


def all_collections():
    page_size, offset, out = 1000, 0, []
    while True:
        page = client.list_all_collections_sort(
            offset=offset,
            limit=page_size,
            sort_column="name",
            ascending=True,
        )
        if not page:
            break
        out.extend(page)
        if len(page) < page_size:
            break
        offset += page_size
    return out


def candidates():
    return [
        c for c in all_collections()
        if c.size_limit is not None and c.size_limit < THRESHOLD
    ]


targets = candidates()
print(f"{len(targets)} collection(s) below {fmt(THRESHOLD)}")

if MODE == "dry-run":
    for c in targets:
        print(f"  {c.id}  {fmt(c.size_limit)} -> {fmt(NEW_LIMIT)}")
    print("\nDry run only. No changes made.")

elif MODE == "pilot":
    if not targets:
        raise SystemExit("Nothing to update.")
    pilot = targets[0]
    client.set_collection_size_limit(collection_id=pilot.id, limit=NEW_LIMIT)
    print(f"Pilot updated: {pilot.id}")
    print("Verify this collection in the UI before running apply.")

elif MODE == "apply":
    failures = []
    for i, coll in enumerate(targets, 1):
        try:
            client.set_collection_size_limit(collection_id=coll.id, limit=NEW_LIMIT)
            print(f"[{i}/{len(targets)}] ok   {coll.id}")
        except Exception as e:
            failures.append((coll.id, str(e)))
            print(f"[{i}/{len(targets)}] FAIL {coll.id}: {e}")

    print(f"\nUpdated {len(targets) - len(failures)}, failed {len(failures)}")
    for cid, err in failures:
        print(f"  {cid}: {err}")

Two properties worth knowing:

• The apply stage re-queries. It builds the target list fresh, so the pilot collection drops out automatically and any collections created since the dry run are included.
• The operation is resumable. If it is interrupted, already-updated collections no longer match the filter, so re-running continues where it stopped rather than repeating work.

Collections with no limit set are reported as unlimited and deliberately skipped, since setting a limit on them would reduce their capacity rather than raise it. To clear a limit instead of raising it, swap in remove_collection_size_limit(collection_id=...), which reverts the collection to the platform default. Consider storage capacity before doing this at scale.

Running this at scale

On deployments with thousands of collections the apply stage makes one API call per collection and will take time.

• Run it during a quiet period rather than at peak usage.
• Capture the output to a file so you have a record of what changed and what failed.
• Individual failures do not stop the run. Review the failure summary at the end and re-run to retry only those, since successful collections drop out of the filter.
• If failures are widespread rather than isolated, stop and investigate instead of re-running — that usually points at a permission or connectivity problem rather than per-collection issues.

How to verify

1. After the pilot, open that collection's settings in the UI and confirm the size limit reads as the new value.
2. After the apply stage, re-run the script in dry-run mode. It should report 0 collections below the threshold.
3. Spot-check several collections in the UI across different owners, not just ones you created.
4. Confirm a user can now upload beyond the old ceiling in a previously affected collection.
5. Revoke the API key created for this task and confirm it no longer appears as active.

If this doesn't resolve it

If the dry run returns far fewer collections than you expect, the account running the script probably cannot see all of them. Confirm it holds collection-management permissions, and compare the count against the total shown on the collections page in the System Dashboard.

If limits still enforce the old value after updating, open a support ticket and include:

• Exact h2oGPTe version and h2oGPTe Python client version
• The count reported by the dry run, and the total number of collections on the deployment
• The role and permissions of the account whose API key was used — not the key itself
• The limit value in bytes that you set, so decimal-versus-binary can be ruled out
• For an affected collection: the limit reported by the API and the limit shown in the UI
• Any errors from the apply stage, with collection IDs

Related

• Collections usage overview — what a collection is and how its settings behave.
• Update a collection's settings — the per-collection route used by the workaround.
• h2oGPTe Python API reference — set_collection_size_limit, remove_collection_size_limit and list_all_collections_sort.
• Account menu — creating, scoping and deleting API keys.
• Roles and permissions — confirming the account can manage all collections.