Skip to main content

GCP - Dataproc -- gcs buckets -- pyspark jobs

Dataproc --> Hadoop managed services available in GCP 

In order to create a cluster go to search data proc inside Google Cloud console search data proc and then enable its related API 






then navigate into the cluster and start creating a new cluster. inorder to create the clusters follow give these commands in the cloud shell 



Before creating the cluster, we need permission to do so. To do that, go to IAM, search for service accounts, select the particular service account, and grant access to create the access.
 in general, unlike AWS (user level) in GCP we create service account according to the entire project.




step1:gcloud compute networks create dataproc-network2 --subnet-mode=auto

inorder to create the cluster network

step 2: gcloud compute firewall-rules create allow-internal --allow all --source-ranges 10.128.0.0/9 --network dataproc-network2

creating network and firewall rules to create communication between the notes to master --> external

step 3: gcloud compute firewall-rules create allow-external2 --allow tcp:22,tcp:3389,icmp --network dataproc-network2

creating network and firewall rules to create communication between the notes to master - internal

step 4:gcloud dataproc clusters create test-cluster2 --region us-central1 --zone us-central1-a --master-machine-type n2-standard-2 --master-boot-disk-size 50 --num-workers 2 --worker-machine-type n2-standard-2 --worker-boot-disk-size 50 --network dataproc-network2 --enable-component-gateway --image-version 2.2.40-debian12 --project dataengineering-jan2025

creating how many nodes and size of the master , region and all


created the cluster 



So in earlier sesission... after creating cluster with mater and worker nodes....to interact with the machine, we went into ssh mode. and entered commands like hdfs dfs ...hive etc when we want to interact with dataproc cluster...

So now with the same dataproc cluster we want to submit the spark jobs - what we can do is, there are some commands to submit jobs....before that we need to place our spark job and any relavant files like  reading local data files...so that place is called gcs buckets in google cloud, in aws we call as s3 buckets, in azure we call as blocks...it is a cloud storage where we place the code we are going to use to run it in cluster.

What are GCS Buckets?

  • Containers for Data:
    • GCS buckets are essentially containers that hold your data objects (files). You can store any type of data in them, from text files and images to large datasets and backups.
  • Global Namespace:
    • Bucket names are globally unique across all of Google Cloud Storage. This means that once a bucket name is taken, no one else can use it.
  • Object Storage:
    • GCS is an object storage service, meaning data is stored as objects within buckets. This differs from file systems that use hierarchical directories.
inoder to create the bucket incase go to cloud storage and create bucket and view what is inside the bucket .below buckets are created when the clusters are created .


creating the bucket for test and do spark jobs and named it as spark_jobs1 and uploaded the already created py file .



uploading the py files and their related files into gcs buckets to test the spark jobs.
In py files, edit the location of the files since the files have been moved into the GCS buckets.





command to run the pyspark the job - gcloud dataproc jobs submit pyspark --cluster name --region /bucket location gs://bujet name/python file name.

eg : gcloud dataproc jobs submit pyspark --cluster=cluster-1bd6 --region=us-central1 "gs://spark_jobs1/Spark Practice.py"



and we can see the results of the jobs and dag under the particular cluster under jobs 





  1. first we wrote spark code- which contains transformations and actions.
  2. we uploaded the spark code into gcs bucket
  3. then submitted the spark job using command - gcloud spark submit...gs://buketname/sparkcode.py
  4. written the output to bucket ...we can also write the output to bigquery aswell..very simple as shown below..


5. if we also want tp wrte the data into something like a database, then we will use format as jdbc..as shown below:..in google search - pyspark write dataframe to postgresql, with username and pwd etc

JDBC (Java Database Connectivity)

JDBC is an API (Application Programming Interface) in Java that allows applications to connect to and interact with databases...







refer to Rama Blogs for reminding 








Labels:

Comments

Post a Comment

Popular posts from this blog

Entity Relationship (ER) Diagram Model with DBMS Example

Reference :   Entity Relationship (ER) Diagram Model with DBMS Example What is ER Diagram? ER Diagram  stands for Entity Relationship Diagram, also known as ERD is a diagram that displays the relationship of entity sets stored in a database. In other words, ER diagrams help to explain the logical structure of databases. ER diagrams are created based on three basic concepts: entities, attributes and relationships. ER Diagrams contain different symbols that use rectangles to represent entities, ovals to define attributes and diamond shapes to represent relationships. At first look, an ER diagram looks very similar to the flowchart. However, ER Diagram includes many specialized symbols, and its meanings make this model unique. The purpose of ER Diagram is to represent the entity framework infrastructure. Entity Relationship Diagram Example Table of Content: What is ER Diagram? What is ER Model? History of ER models Why use ER Diagrams? Facts about ER Diagram Model ER Diagram...
Post a Comment
Read more

SQL Joins and advanced joins and Subqueries

  Refernce :  Expert Guide to Advanced SQL Joins: What You Need to Know It's helpful to visualize how these different SQL joins work. Here's a breakdown in a table-like format, along with explanations: SQL Join Types Overview Join Type Description Key Characteristics Use Cases INNER JOIN Returns rows where there is a match in both tables. - Shows only matching records. - Excludes unmatched rows from both tables. - Retrieving related data that exists in both tables. - Finding records with corresponding entries in another table. LEFT OUTER JOIN (LEFT JOIN) Returns all rows from the left table, and the matched rows from the right table. - Includes all records from the left table. - Fills in NULL values for columns from the right table where there's no match. - Retrieving all records from one table and their related data from another, even if some records don't have matches. - Finding records in one table that don't have corresponding entries in another. RIGHT OUTER JO...
Post a Comment
Read more

GIT BASH

  Bash Shell: Git Bash uses the Bash (Bourne Again SHell) command-line interpreter. This means you can use many of the same commands you'd find in a Linux or macOS terminal. Git Integration: Git Bash is tightly integrated with Git, making it easy to execute Git commands Essential Commands: Navigation: pwd : Prints the current working directory. ls : Lists files and directories in the current directory. cd <directory> : Changes the current directory. cd .. : Moves to the parent directory. File Management: mkdir <directory> : Creates a new directory. touch <file> : Creates a new file. rm <file> : Removes a file. rmdir <directory> : Removes an empty directory. Git Commands: git init : Initializes a new Git repository. git clone <repository URL> : Clones an existing Git repository. git status : Displays the status of your working directory. git add <file> : Adds a file to the staging area. git commit -m "commit message" : Commits chan...
Post a Comment
Read more