When an upstream task returns a list, a downstream task can map over that list, spawning an independent task instance for each element.
[core] xcom_backend = my_project.xcom_backend.ExclusiveRedisXCom
To expand this to your exclusive business keys, configure custom sensitive patterns in airflow.cfg : airflow xcom exclusive
from datetime import datetime from airflow import DAG from airflow.operators.python import PythonOperator def push_function(**kwargs): kwargs['ti'].xcom_push(key='model_accuracy', value=0.94) def pull_function(**kwargs): ti = kwargs['ti'] accuracy = ti.xcom_pull(task_ids='push_task', key='model_accuracy') print(f"Model accuracy is accuracy") with DAG('legacy_xcom_dag', start_date=datetime(2026, 1, 1), schedule=None) as dag: push_task = PythonOperator(task_id='push_task', python_callable=push_function) pull_task = PythonOperator(task_id='pull_task', python_callable=pull_function) push_task >> pull_task Use code with caution. The TaskFlow API Approach
Think of an XCom as a small note that one task writes down and another task reads later. By default, Airflow saves these notes in its metadata database. Key Terms to Know : Sending data from a task to the database. Pull : Fetching data from the database into a task. Key : The specific name or label given to the shared data. How XCom Works by Default When an upstream task returns a list, a
By default, xcom_pull scopes its search to the current DAG run. However, dynamic tasks or poorly configured task dependencies can inadvertently read historical XCom data. To enforce exclusivity per execution loop, always rely on explicit execution date contexts:
You push a result, but no downstream task is allowed to pull it. Solution: Define the exclusive mapping at DAG level, and review with airflow dags show-xcom --exclusive-violations . By default, Airflow saves these notes in its
def load(**context): final = context['ti'].xcom_pull(task_ids='transform') print(final)
XCom is exclusive to . Do not use it to pass large datasets between tasks; instead, write the large data to a file in cloud storage (S3/GCS) and pass the file path via XCom to the next task.
XComs allow tasks to exchange small amounts of metadata. Unlike a traditional data bus or a shared memory space, XComs operate via a pull-and-push mechanism explicitly recorded in Airflow's metadata database. The Storage Mechanism
is the standard for Airflow 2.0+ for cleaner, implicit XCom handling.