SQL analysis runs on the Airflow worker
AWS announced native DuckDB operator support for Amazon Managed Workflows for Apache Airflow on November 9. The announcement covers MWAA Provisioned environments running Apache Airflow 3.3.1; the current guide describes native support for that version and later.
DuckDB runs inside an Airflow task on the environment’s worker. It can perform checks, summaries and transformations on supported data, including files in Amazon S3, without requiring a separate database server or cluster. This does not establish a performance gain or a particular cost saving.
Already running 3.3.1 does not always mean the image is current
The guide says a 3.3.1 environment created before DuckDB support became available needs an environment update to apply the latest MWAA image. AWS instructs users to leave environment settings unchanged for that update and notes that it can take time.
On the current image, the DuckDBExecuteQueryOperator can use the default duckdb_default connection. S3 reads and writes use the environment’s execution role; access to the required data stores still has to be granted. An existing connection with that name is not automatically overwritten.
Task state, extensions and older versions have boundaries
Each task starts with an empty in-memory database. To pass results between tasks, AWS directs users to write them to S3. Queries share worker memory with other tasks, so native support does not remove memory constraints.
For Airflow 2.11 through 3.2, the guide describes manually adding DuckDB and its provider. Airflow 2.10 and earlier are outside the stated provider requirement. Installing a different DuckDB version can also require matching S3 extensions, including separate preparation where internet access is unavailable. Check the linked guide against the actual environment before changing it.
