Apify integration

What it does

The Apify integration lets your agency run web scrapers for a client from inside a workflow without operating any scraping infrastructure yourself. Apify runs the Actor on its own machines, through its own proxies, and hands back structured JSON. Your workflow just starts the run and reads the results.

Connect an Apify account once and your workflows can start any Actor from the Apify Store or your own account, run a saved task, scrape a single page to text or markdown, read scraped records out of a dataset, inspect or list past runs, read and write key-value store records, and abort a run that is overrunning its budget. Two triggers fire when an Actor or task run finishes, so a scrape that takes twenty minutes does not have to hold a workflow open while it works.

Connect an Apify account

1. Open your workspace in TaskJuice and navigate to Connections.
2. Choose Apify and click Connect.
3. In a new tab, open Apify Console and go to Settings, then API & Integrations.
4. Create an API token and copy it.
5. Paste the token into TaskJuice. Apify authenticates every request with Authorization: Bearer {token}.

Prefer a scoped token over a full-access one. Grant it Run permission on the specific Actors you automate plus read access to their datasets, and set an expiry date. Scoped tokens cannot create or modify Actors, which none of these actions do.

To rotate a token, regenerate it in Apify Console and update the TaskJuice connection. The previous token keeps working for 24 hours, so you can swap it over without an outage.

Who owns the account

Apify is a developer platform, so in almost every engagement the agency owns the Apify account and the client never has one. That makes Apify a good fit for an account-scoped connection granted down to each client workspace, rather than a credential you collect from the client.

One consequence is worth planning for: Apify bills compute units to the account that owns the token, not to the client whose workflow triggered the run. Set a monthly spend cap on each connection grant, and set the Max charge field on any client-facing run, so a single misconfigured scrape cannot drain the shared budget.

Triggers

• apify/run-succeeded fires when a run finishes successfully. The payload carries the full run object, including resource.defaultDatasetId, which you pass straight to Get Dataset Items.
• apify/run-failed fires when a run ends without succeeding, covering failed, aborted, and timed-out runs. Read eventType to tell the three apart. Use it to alert on a client's overnight scrape breaking rather than discovering the gap later.

Both triggers fire for Actor runs and saved task runs alike. Apify reports both through the same ACTOR.RUN.* event family, so a run started from a task is distinguished only by carrying an actorTaskId. Branch on that field if a workflow needs to tell them apart.

Both triggers publish an inbound URL. In Apify Console, open the Actor or task, add a webhook, paste the URL, and choose which run events should fire it. Apify does not sign webhook bodies, so also add a header in the webhook's Headers template carrying a shared secret, and paste that same secret into the TaskJuice trigger. TaskJuice verifies the header on every inbound POST before activating the workflow.

Apify retries a failed delivery with an expanding backoff that runs out to roughly 32 hours, and states that a webhook may be delivered more than once. Build the workflow so a repeated delivery is harmless.

Actions

• apify/run-actor starts an Actor and returns the run object immediately, without waiting for the scrape to finish. This is the action to use for anything long.
• apify/run-actor-and-get-dataset-items starts an Actor, waits for it, and returns the dataset items in one step.
• apify/run-task starts a saved Actor task using the input already configured in Apify, with an optional per-run override.
• apify/scrape-single-url scrapes one page with Apify's Website Content Crawler and returns its content as text, markdown, or HTML.
• apify/get-dataset-items reads scraped records out of a dataset, with field projection, offset paging, and Return All / Limit auto-pagination.
• apify/get-run looks up a single run by ID, including its status, its cost so far, and the storages holding its results.
• apify/get-last-run fetches the most recent run of an Actor, optionally filtered by status.
• apify/get-user-runs lists runs across the account, filtered by status or start date, with Return All / Limit auto-pagination.
• apify/get-key-value-store-record reads one record from a key-value store, such as an Actor's INPUT.
• apify/set-key-value-store-record writes or overwrites a record in a key-value store.
• apify/abort-run stops a run that is still starting or running, optionally letting it persist state first.

Return All and Limit

List actions that page (get-dataset-items, get-user-runs) carry a Return All toggle and a Limit field in their collapsed Pagination section:

• Return All fetches every page and returns the complete result set. The platform caps a Return All read at the MAX_RETURN_ALL_ITEMS ceiling of 10,000 items; a read that would exceed it fails loudly with a RESULT_TRUNCATED error instead of returning a silently partial result. Narrow the query, or set a Limit to accept the first N items.
• Limit fetches only the first N items (1–10,000) and succeeds with exactly that many when more exist. It is ignored while Return All is on — the panel greys it out.
• With neither set, the action returns a single page, exactly as before.

Paged object responses keep their declared envelope shape: the concatenated items replace the page's item array, and sibling fields (paging counters, links) reflect the final request.

Get Dataset Items returns a flat array (its natural shape), so Return All simply returns the full concatenated array.

Handling a long scrape

Run Actor and Get Dataset Items is the convenient option, but Apify ends the wait at 300 seconds and returns a timeout while the run carries on in the background. Retrying does not recover the original run, it starts a second one you also pay for. Use it only for scrapes you are confident finish in a couple of minutes.

For anything longer, split the work:

1. Start the scrape with Run Actor.
2. Let the workflow finish, or park it on a Wait step.
3. Resume from apify/run-succeeded when Apify reports the run completed.
4. Read the results with Get Dataset Items, using resource.defaultDatasetId from the trigger payload.

If you park the workflow on a Wait step, put that step's resume URL into the Ad-hoc webhooks field on Run Actor. Apify then calls back to exactly that waiting workflow when the run finishes, so a twenty-minute scrape costs you nothing while it runs.

Known limitations

• The synchronous run actions end their wait at 300 seconds and return a timeout while the run continues in the background. The Wait for finish option on the async actions is a different mechanism and Apify caps it at 60 seconds.
• Apify bills compute units to the account that owns the token, so per-client cost attribution has to be handled with connection grant spend caps and the Max charge field rather than by Apify.
• Datasets that were never given a name expire after seven days. A workflow that revisits a dataset later should reference a named one.
• Webhook bodies are not signed. Verification relies on the shared secret header you configure in the webhook's Headers template.
• Deliveries are at-least-once and retried for up to roughly 32 hours, so triggers should be treated as repeatable.
• The API allows 60 requests per second per resource, raised to 400 for run operations, and returns a rate-limit error above that.
• Actors are third-party programs. An Actor can change its input shape or break when the site it scrapes changes, independently of this integration.