Skip to main content

Resumable Activity (AKA Pause On Failure)

TLDR

After retries are exhausted, park the Workflow in a waiting state and block on a Signal that notifies the Workflow to proceed or optionally delivers corrected input from a human operator, then re-execute the Activity. Use this when failures are caused by bad input that can be fixed externally — so the Workflow resumes exactly where it left off instead of being restarted from scratch.

Overview​

The Resumable Activity pattern parks a Workflow, durably waiting, after Activity retries are exhausted, waits for a corrective Signal from a human operator, then re-executes the Activity with the corrected input. Use it when failures are caused by bad input data that can be corrected externally — a wrong account number, an invalid reference, a missing record that will be created — and abandoning the Workflow is worse than pausing it.

Problem​

When an Activity fails due to bad input, retrying with the same input will never succeed. The standard options are:

  • Fail the Workflow immediately. The client must restart the process from scratch, often re-entering the same data that caused the failure.
  • Mark the error as non-retryable. Same result — the Workflow fails and the client has no way to inject a correction.
  • Poll for the correction from inside the Workflow. Wastes resources for a fix that may never come.

What you actually want is for the Workflow to pause — consuming zero resources — until an authorized operator provides corrected data, then resume exactly where it left off. Temporal's durable execution model makes this possible without any external database, queue, or polling mechanism.

Solution​

Use a bounded RetryPolicy to allow a few automatic retries (in case the failure is transient), then catch the exhausted ActivityError in the Workflow. Transition to an AWAITING_CORRECTION state and block on workflow.wait_condition (or equivalent). Register a Signal handler that accepts the corrected input and unblocks the condition. When the Signal arrives, re-execute the Activity with the corrected input. A second Signal gates the final approval before completing.

The following describes each step:

  1. The client starts the Workflow with an invalid account number.
  2. The Activity fails. Temporal retries automatically up to the configured maxAttempts.
  3. When retries are exhausted, the Workflow catches the ActivityError and sets its status to AWAITING_CORRECTION.
  4. The Workflow parks itself using wait_condition — it consumes no CPU, no polling, no timers. Its state is fully persisted in Temporal.
  5. An admin notices the problem (via Temporal UI, an alert, or an operations dashboard) and sends a retryWithCorrection Signal with the corrected account number.
  6. The Workflow wakes up, applies the correction, and re-executes the Activity — which now succeeds.
  7. The Workflow transitions to AWAITING_APPROVAL and parks again, waiting for the client to approve the transfer.
  8. The client sends an approve Signal. The Workflow completes and returns the result.

The key insight: the Workflow never died. It survived bad input, waited indefinitely without polling, accepted an external correction, and completed cleanly. Its entire state — status, corrected account, approval decision — is durable in Temporal throughout.

Implementation​

Workflow with correction and approval signals​

The Workflow maintains state as named fields. Signal handlers set the fields, and wait_condition blocks until they are non-null.

# workflows.py
from dataclasses import dataclass
from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy, SearchAttributeKey
from temporalio.exceptions import ActivityError
import activities

TRANSFER_STATUS_KEY = SearchAttributeKey.for_keyword("TransferStatus")

@dataclass
class TransferInput:
    from_account: str
    to_account: str
    amount: float

@workflow.defn
class TransferWorkflow:
    def __init__(self) -> None:
        self._status = "PENDING"
        self._corrected_account: str | None = None
        self._approval: bool | None = None

    @workflow.run
    async def run(self, transfer: TransferInput) -> str:
        account = transfer.to_account
        correction_attempts = 0

        while True:
            self._status = "TRANSFERRING"
            try:
                result = await workflow.execute_activity(
                    activities.execute_transfer,
                    TransferInput(transfer.from_account, account, transfer.amount),
                    start_to_close_timeout=timedelta(seconds=30),
                    retry_policy=RetryPolicy(maximum_attempts=3),
                )
                break  # Activity succeeded — exit the correction loop
            except ActivityError:
                correction_attempts += 1
                if correction_attempts > 5:
                    self._status = "FAILED"
                    workflow.upsert_search_attributes([TRANSFER_STATUS_KEY.value_set(self._status)])
                    raise
                self._status = "AWAITING_CORRECTION"
                workflow.upsert_search_attributes([TRANSFER_STATUS_KEY.value_set(self._status)])
                workflow.logger.warning(
                    "Transfer failed — waiting for account correction",
                    extra={"to_account": account},
                )
                # Park until the admin sends a correction signal
                await workflow.wait_condition(
                    lambda: self._corrected_account is not None
                )
                account = self._corrected_account
                self._corrected_account = None

        self._status = "AWAITING_APPROVAL"
        await workflow.wait_condition(lambda: self._approval is not None)

        if self._approval:
            self._status = "COMPLETED"
            return f"Transfer of {transfer.amount} to {account} completed"
        self._status = "REJECTED"
        return "Transfer rejected by client"

    @workflow.signal
    def retry_with_correction(self, corrected_account: str) -> None:
        self._corrected_account = corrected_account

    @workflow.signal
    def approve(self, approved: bool) -> None:
        self._approval = approved

    @workflow.query
    def get_status(self) -> str:
        return self._status

Sending signals​

An operator sends the correction Signal using the Temporal CLI or any SDK client. The Workflow wakes immediately when the Signal is delivered.

# Correct the account number
temporal workflow signal \
  --workflow-id transfer-wf-001 \
  --name retryWithCorrection \
  --input '"account-123"'

# Approve the transfer
temporal workflow signal \
  --workflow-id transfer-wf-001 \
  --name approve \
  --input 'true'

Activity implementation​

The executeTransfer Activity must distinguish between permanent failures — such as an invalid account number — and transient failures that Temporal should retry automatically. Throw a non-retryable ApplicationFailure for permanent input errors so the Workflow catches the ActivityError immediately and transitions to AWAITING_CORRECTION instead of exhausting all retry attempts first. Let all other exceptions propagate so the RetryPolicy handles transient failures.

# activities.py
from temporalio import activity
from temporalio.exceptions import ApplicationError

@activity.defn
async def execute_transfer(transfer: TransferInput) -> str:
    # Non-retryable: bad account number requires a human correction, not a retry.
    if not await account_service.exists(transfer.to_account):
        raise ApplicationError(
            f"Account {transfer.to_account} not found",
            type="AccountNotFoundError",
            non_retryable=True,
        )
    # Other exceptions propagate as retryable so the RetryPolicy handles them.
    return await payment_service.transfer(
        transfer.from_account, transfer.to_account, transfer.amount
    )

State Diagram​

The Workflow transitions through a well-defined set of states. Query getStatus at any time to observe the current state.

Best practices​

  • Use a bounded MaximumAttempts before parking. Allow a few automatic retries to recover from transient failures. Parking immediately on the first failure forces operators to intervene for problems that would have resolved on their own.
  • Manage history growth in long-running correction loops. Each correction cycle — park, receive signal, re-execute Activity — adds events to the Workflow history (signal received, state transitions, Activity scheduled/completed). For workflows that may receive many corrections over time, use Continue-As-New to carry the current state into a fresh execution before the history grows too large, rather than relying solely on an arbitrary correction counter.
  • Expose status via a Query method. The getStatus Query gives operations tooling visibility into where the Workflow is parked without requiring access to the Workflow history.
  • Validate the correction in the Signal handler. Check that the corrected account is non-empty and matches the expected format before setting the state. An invalid correction just parks the Workflow again, but a clear error message helps operators.
  • Log and record state at every transition. The AWAITING_CORRECTION and AWAITING_APPROVAL states can last hours or days. Structured log lines at each transition make the audit trail clear. For operational visibility, also update a Search Attribute at each transition (for example, a Keyword attribute storing the current status) so operators can filter and query workflows by state directly from the Temporal UI or CLI.
  • Notify the operator proactively. The AWAITING_CORRECTION transition is a good point to send an alert — an email, a Slack message, or a ticket — rather than waiting for the operator to notice in the Temporal UI.
  • Distinguish this from the Approval pattern. The Approval pattern gates forward progress on a human decision. This pattern recovers from failure with a human-supplied data correction. Both use Signals and wait_condition, but serve different roles in a process.

Common pitfalls​

  • Waiting without a timeout. If operators never send the correction Signal, the Workflow waits indefinitely. Add durable timer if the process must resolve within a time bound.
  • Not clearing the correction state before re-entering the loop. After applying the correction, set corrected_account = None (or equivalent) before the next Activity attempt. Otherwise, if the corrected activity also fails, the Workflow immediately re-uses the previous correction instead of waiting for a new one.
  • Accepting corrections in the wrong state. If a Signal arrives while the Activity is running (not parked), the correction should be queued and applied after the current attempt completes. The Signal handler always runs — the condition check (wait_condition) determines when the Workflow acts on it.
  • Conflating the correction loop with a general retry loop. This pattern is for correcting input data. For retrying the same call against a temporarily unavailable system, use Fast/Slow Retries instead.