Skip to main content

Posts

SQL for data engineer

These are excellent SQL interview habits. Here's a polished version you can use as your SQL Problem-Solving Checklist during interviews. SQL Interview Mindset Checklist 1. Don't select unnecessary columns ❌ Bad SELECT * FROM employees; ✅ Good SELECT employee_id, employee_name, salary FROM employees; Why? Improves query performance. Reduces data scanned (especially in BigQuery, where you pay for data processed). Makes the query easier to read. Shows the interviewer you understand optimization. Interview Tip: "I avoid SELECT * unless I'm exploring the data. In production, I select only the required columns." 2. Ask clarifying questions whenever you're unsure Don't make assumptions. Examples: What does this column represent? Is this table already cleaned? Are duplicate records possible? Can one customer have multiple orders? Should NULL values be included? Should cancelled orders be considered? Do we need the latest record or all records? What...

GCP - Basics

  Google Cloud Resources: Global, Regional, and Zonal Google Cloud resources are generally classified into three categories based on their scope: Global Resources Regional Resources Zonal Resources 1. Global Resources Global resources are not limited to a specific region or zone. They are accessible across all Google Cloud regions. Examples of global resources include: VPC Networks IAM Cloud DNS Firewall Rules Global Load Balancers A VPC network is a global resource, meaning it can span multiple regions. 2. Regional Resources A region is a geographic location where Google Cloud operates one or more data centers. This is where you deploy your cloud infrastructure. Each region contains multiple zones , providing high availability and fault tolerance. Examples of regional resources include: Subnets Cloud SQL Cloud NAT Cloud Routers Regional Persistent Disks A subnet is always regional , meaning it belongs to a single region. Multiple subnets can exist within the same global VPC, wit...

3.Python

 In last class , I learned about API request library and how to get the data , now i can going to do the same with IDE (PyCharm ). step 1 : importing request library step 2 : extracting the data and exploring inside that  step 3 : doing transformations using pandas lib  step 4 : write it into csv file  this is normal ETL process. python program : Always write the program using functions to look the code clean function block then main code  main block : if __name __ = "__main__": exmaple : def get_input():     num = int(input("Enter a number: "))     return num def display_square(num):     print("Square =", num * num) def main():     number = get_input()     display_square(number) if __name__ == "__main__":     main()

2.a flattening the JSON File.

Think of JSON processing as this journey: Raw JSON → Python objects → Flatten → Clean → DataFrame → Spark → Production pipeline Goal In Local Python In PySpark (Databricks) In Cloud SQL (Snowflake/BigQuery) Read a file json.load(f) spark.read.json() COPY INTO / Storage Integration Go inside an object data["key"]["subkey"] df.select("key.subkey") SELECT column:key.subkey Turn a list into rows for item in my_list: explode(col("my_list")) LATERAL FLATTEN() / UNNEST() Phase 1 — JSON Fundamentals 1. JSON Data Types You need to immediately recognize how JSON maps to Python. JSON Python Example Object dict {"name":"John"} Array list [1,2,3] String str "London" Number int/float 100 Boolean bool true Null None null Example: { "employee":{ "id":100, "name":"John" }, "skills":[ "Python", "Spark" ] } Python sees this as: { "employee...