Skip to main content

Composer - GCP




the DAG files are stored within a "dags" folder, which resides inside of a Cloud Storage bucket, that is created when a Cloud Composer environment is created.




inside the DAG file there will be the default py file for orchestration.

you can create the number of DAG files and upload into this folder to create the workflow

if you click on the files, it will lead to airflow monitoring under graph you can see DAG and operator (task) , and under code ,you can see the code for orchestration 



if the DAG is yellow in color that means it's not working properly then we have to debug if its in green color its works properly .


from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime, timedelta

default_args = {
    'owner': 'your_name',
    'start_date': datetime(2023, 10, 26),
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
    'dagrun_timeout': timedelta(hours=2),
}

with DAG(
    dag_id='my_data_pipeline',  # DAG Name
    schedule_interval='@daily',  # Schedule interval
    default_args=default_args,
    catchup=False, The catchup parameter determines whether Airflow should "catch up" on past scheduled runs that were missed.
) as dag:
    task1 = BashOperator(
        task_id='extract_data',  # Task ID
        bash_command='echo "Extracting data..."',
    )

    task2 = BashOperator(
        task_id='transform_data',  # Task ID
        bash_command='echo "Transforming data..."',
    )

    task1 >> task2  # Define task dependencies, >> operator defines the order in which tasks should be executed

Key Points:

  • DAG parameters control the overall behavior of the workflow.
  • Operator parameters define the individual steps within the workflow.
  • task_id values must be unique within a DAG.
  • The dag_id must be unique across all dags inside of the airflow environment.
  • The schedule_interval is what tells airflow when to run the dag.
  • The dagrun_timeout is a safety net.

I hope this helps clarify these important Airflow concepts.

Labels:

Comments

Post a Comment

Popular posts from this blog

Entity Relationship (ER) Diagram Model with DBMS Example

Reference :   Entity Relationship (ER) Diagram Model with DBMS Example What is ER Diagram? ER Diagram  stands for Entity Relationship Diagram, also known as ERD is a diagram that displays the relationship of entity sets stored in a database. In other words, ER diagrams help to explain the logical structure of databases. ER diagrams are created based on three basic concepts: entities, attributes and relationships. ER Diagrams contain different symbols that use rectangles to represent entities, ovals to define attributes and diamond shapes to represent relationships. At first look, an ER diagram looks very similar to the flowchart. However, ER Diagram includes many specialized symbols, and its meanings make this model unique. The purpose of ER Diagram is to represent the entity framework infrastructure. Entity Relationship Diagram Example Table of Content: What is ER Diagram? What is ER Model? History of ER models Why use ER Diagrams? Facts about ER Diagram Model ER Diagram...
Post a Comment
Read more

GCP Buckets

  CREATING BUCKET Method 1: Using GCP Console (UI) Go to the  Google Cloud Console . Navigate to  Cloud Storage . Click  Create Bucket . Enter a  unique name  for your bucket. Select a  storage class  (Standard, Nearline, Coldline, or Archive)... Choose a  location  (region/multi-region). Set  access control  (Uniform or Fine-grained). Click  Create . how do you optimise the class storage buckets - we can talk about cloud storage choosing wisely standard vs nearline vs coldline vs archive... how you managed your buckets - how did you manage? life cycle rule - regions multi regions etc
Post a Comment
Read more

Pyspark

pyspark link the above link for a sample code for Pyspark  Transformations create new datasets, actions return values, or write data. Transformations: Definition: Transformations are operations that create a new RDD or data frame from an existing one. They are "lazy," meaning they don't execute immediately. Instead, Spark builds a lineage graph (DAG - Directed Acyclic Graph) of transformations. Examples: map() : Applies a function to each element. filter() : Select elements based on a condition. flatMap() : Similar to a map, but flattens the results. groupByKey() : Groups elements by key. reduceByKey() : Reduces elements by key. join() : Joins two datasets. distinct() : Removes duplicate elements. union() : Combines two datasets. coalesce() : Reduces the number of partitions. repartition() : Increases or decreases the number of partitions. Actions: Definition: Actions trigger the execution of the lineage graph, which returns a result to the driver progra...
Post a Comment
Read more