Skip to main content

factors to considers for choosing the storage account in cloud

 Type of data that is to be stored 

Structured 

semi-structured 

unstructured

The factors depend on operational needs

1. How often is the data accessible?

2. How quickly do we need to serve?

3. Need to run simple Queries?

4. Need to run complex queries?

5. accessed from Multiple regions?


1. Azure Storage Account: Think of an Azure Storage Account as the top-level resource that groups together various Azure Storage services. When you create a "General-purpose v2" storage account, it can contain:

  • Azure Blob Storage: For unstructured data (files, images, videos, backups).

  • Azure File Storage: For managed file shares (SMB/NFS).

  • Azure Queue Storage: For messaging (queues).

  • Azure Table Storage: For NoSQL key-value data.

  • Azure Disks: For VM disks (though often managed separately as "Managed Disks").

2. Azure Blob Storage:

  • This service is for storing unstructured data (Binary Large Objects or "blobs").

  • It's like a vast digital bucket where you put files. These files are accessed via HTTP/HTTPS endpoints.

  • It has different "types of blobs" (Block, Page, Append) and "access tiers" (Hot, Cool, Archive) as we discussed, but none of these are "queues."

3. Azure Queue Storage (The Actual Queue Service):

  • Purpose: Azure Queue Storage is a service specifically designed for storing large numbers of messages. Its primary use case is to decouple components of an application, enabling asynchronous communication.

  • How it works:

    • One part of an application (the "producer") places messages into a queue.

    • Another part of the application (the "consumer" or "worker") retrieves and processes these messages at its own pace.

    • This makes applications more scalable, resilient, and responsive.

  • Key Characteristics:

    • Messages: Each message can be up to 64 KB in size.

    • Retention: Messages can remain in the queue for up to 7 days by default (and can be set to never expire for newer API versions).

    • Access: Messages are typically accessed on a First-In, First-Out (FIFO) basis, though due to its distributed nature, strict order isn't guaranteed without specific design patterns.

    • Visibility Timeout: When a message is retrieved, it becomes "invisible" to other consumers for a configurable period (visibility timeout), preventing multiple workers from processing the same message simultaneously. If processing fails, the message reappears after the timeout.

    • Scalability: Can store millions of messages, up to the total capacity limit of the storage account.

Azure Blog  Gen2 storage 
 Azure Delta Lake Gen2 refers to the powerful synergy between two key Azure components for modern data analytics:

  1. Azure Data Lake Storage Gen2 (ADLS Gen2): This is Microsoft's enterprise-grade, highly scalable, and cost-effective data lake solution built on top of Azure Blob Storage. Its key differentiator is the Hierarchical Namespace (HNS), which allows it to organize data into folders and subfolders, much like a traditional file system. This HNS is critical for the performance and compatibility with big data analytics engines like Apache Spark and Hadoop, which expect a file system-like interaction.

  2. Delta Lake: This is an open-source storage layer that brings ACID transactions (Atomicity, Consistency, Isolation, Durability) to data lakes. It was originally developed by Databricks and operates on top of existing cloud object storage (like ADLS Gen2). Delta Lake fundamentally transforms a traditional data lake into a "Lakehouse" architecture, combining the flexibility and low cost of a data lake with the reliability and performance typically found in data warehouses.

What is Azure Delta Lake Gen2?

When we talk about "Azure Delta Lake Gen2," we're referring to implementing Delta Lake as the storage format and transaction layer directly on files stored in Azure Data Lake Storage Gen2.

Essentially:

  • ADLS Gen2 provides the foundational, massively scalable, and cost-effective storage for your raw data files (typically Parquet, ORC, JSON, CSV).

  • Delta Lake sits on top of these files, adding a transactional layer (via a transaction log) and metadata management that provides features traditionally associated with databases.


Characteristics of Delta Lake :

1.ACID Transactions
2. Schema Enforcement and Evolution
3.Time Travel (Data Versioning)
4.Unified Batch and Streaming:
5.Performance Optimizations
6.Full Compatibility with Apache Spark
7.Cost-Effective Storage


Comments

Post a Comment

Popular posts from this blog

Entity Relationship (ER) Diagram Model with DBMS Example

Reference :   Entity Relationship (ER) Diagram Model with DBMS Example What is ER Diagram? ER Diagram  stands for Entity Relationship Diagram, also known as ERD is a diagram that displays the relationship of entity sets stored in a database. In other words, ER diagrams help to explain the logical structure of databases. ER diagrams are created based on three basic concepts: entities, attributes and relationships. ER Diagrams contain different symbols that use rectangles to represent entities, ovals to define attributes and diamond shapes to represent relationships. At first look, an ER diagram looks very similar to the flowchart. However, ER Diagram includes many specialized symbols, and its meanings make this model unique. The purpose of ER Diagram is to represent the entity framework infrastructure. Entity Relationship Diagram Example Table of Content: What is ER Diagram? What is ER Model? History of ER models Why use ER Diagrams? Facts about ER Diagram Model ER Diagram...
Post a Comment
Read more

SQL Joins and advanced joins and Subqueries

  Refernce :  Expert Guide to Advanced SQL Joins: What You Need to Know It's helpful to visualize how these different SQL joins work. Here's a breakdown in a table-like format, along with explanations: SQL Join Types Overview Join Type Description Key Characteristics Use Cases INNER JOIN Returns rows where there is a match in both tables. - Shows only matching records. - Excludes unmatched rows from both tables. - Retrieving related data that exists in both tables. - Finding records with corresponding entries in another table. LEFT OUTER JOIN (LEFT JOIN) Returns all rows from the left table, and the matched rows from the right table. - Includes all records from the left table. - Fills in NULL values for columns from the right table where there's no match. - Retrieving all records from one table and their related data from another, even if some records don't have matches. - Finding records in one table that don't have corresponding entries in another. RIGHT OUTER JO...
Post a Comment
Read more

GIT BASH

  Bash Shell: Git Bash uses the Bash (Bourne Again SHell) command-line interpreter. This means you can use many of the same commands you'd find in a Linux or macOS terminal. Git Integration: Git Bash is tightly integrated with Git, making it easy to execute Git commands Essential Commands: Navigation: pwd : Prints the current working directory. ls : Lists files and directories in the current directory. cd <directory> : Changes the current directory. cd .. : Moves to the parent directory. File Management: mkdir <directory> : Creates a new directory. touch <file> : Creates a new file. rm <file> : Removes a file. rmdir <directory> : Removes an empty directory. Git Commands: git init : Initializes a new Git repository. git clone <repository URL> : Clones an existing Git repository. git status : Displays the status of your working directory. git add <file> : Adds a file to the staging area. git commit -m "commit message" : Commits chan...
Post a Comment
Read more