Skip to content
Back to

Download Files Node

Overview

Section titled “Overview”

Download Files fetches documents from the web and keeps a folder of your Workspace in step with them. You give it sources (a list of file URLs, a Git archive URL, or a sitemap) and a target folder. Each run makes the folder match the sources: new and changed files are written, files that disappeared upstream are removed, and an unchanged source costs one lightweight check instead of a download. Files land as ordinary Workspace files, so everything downstream, from agents to the knowledge graph to other workflows, sees them like any file you uploaded yourself.

URLs can also arrive on a wire: any incoming table with a url column adds its rows to the configured sources (Wires and named tables).

Sources

Section titled “Sources”
  • File URLs: one imported file per URL, HTTPS only.
  • Git archive: a repository archive URL (.tar.gz or .zip, for example a GitHub archive/refs/heads/main.tar.gz link). The archive is unpacked, and the single top-level folder those archives add is stripped. Public archives only.
  • Sitemap: a sitemap or sitemap-index URL. One file per listed page, with bounds you can set: maximum URL count, index recursion depth, per-request timeout, and fetch concurrency. robots.txt is honoured.

Filtering

Section titled “Filtering”

Include and Exclude path patterns apply after a Git archive or sitemap is expanded. By default, repository scaffolding (licences, readmes, CI and editor configuration, lockfiles, build scripts) is excluded, so an imported repository contributes its documents rather than its plumbing. Set your own exclude list to override the default, or an empty one to switch it off.

The target folder

Section titled “The target folder”

The node mirrors its target folder, and the editor warns you when the chosen folder already holds files. The node only ever removes files it wrote itself: a file you or an agent add to the folder by hand is never deleted by a scheduled run. If you remove every source, the node leaves the folder untouched rather than emptying it. The folder must be a real folder; the Workspace root is not a valid target.

Keeping it current

Section titled “Keeping it current”

Add a schedule to the node (see Node triggers) and the import re-runs on cron. Every run first asks each source whether it changed, using the origin’s own validators, and skips the download when nothing moved: a daily mirror of an unchanged archive costs one conditional request. A manual Run behaves the same way and re-fetches exactly when a source changed.

Credentials

Section titled “Credentials”

A URL can reference a Workspace secret as {{secret.NAME}} (see Workspace secrets). For sources that need authentication, set an Authorization header — it is the only request header the node sends, and its value can also be a {{secret.NAME}} reference. Credentials are sent only to the origin you configured and are dropped if a redirect leaves it. Secret values never appear in logs or error messages.

Output

Section titled “Output”

The node’s output port lists the files the run wrote, one row per file with its path, size and content hash, so downstream nodes can react to what arrived. The files themselves live in the target folder.

Limits

Section titled “Limits”
  • HTTPS sources only; private network addresses are refused.
  • A single file larger than 200 MB cannot be imported.
  • Archive unpacking enforces size and file-count caps and rejects unsafe archive entries.