Skip to main content

Python - Pandas lib

Pandas is a Python library.

Pandas are used to analyze data.


A Pandas Series is like a column in a table.
a = [1, 7, 2]

myvar = pd.Series(a)

print(myvar[0]) --> output will be 1 (or)

myvar = pd.Series(a, index = ["x", "y", "z"])

Labels

If nothing else is specified, the values are labeled with their index number. First value has an index 0, second value has index 1 etc.

This label can be used to access a specified value.

if its dict to data frame 

calories = {"day1": 420, "day2": 380, "day3": 390}

myvar = pd.Series(calories)

print(myvar)

output :

day1    420
day2    380
day3    390

DataFrames

Data sets in Pandas are usually multi-dimensional tables, called DataFrames.

A series is like a column, a DataFrame is a whole table.

data = {
  "calories": [420, 380, 390],
  "duration": [50, 40, 45]
}

myvar = pd.DataFrame(data)

print(myvar)

Pandas use the loc attribute to return one or more specified row(s)

print(df.loc[0])

  calories    420
  duration     50
  Name: 0, dtype: int6
to_string() to print the entire DataFrame.

1. Importing pandas:

Python
import pandas as pd
import numpy as np # Often used with pandas

2. Creating DataFrames:

  • From a dictionary:
Python
data = {'col1': [1, 2, 3], 'col2': ['a', 'b', 'c']}
df = pd.DataFrame(data)
  • From a list of lists:
Python
data = [[1, 'a'], [2, 'b'], [3, 'c']]
df = pd.DataFrame(data, columns=['col1', 'col2'])
  • From a CSV file:
Python
df = pd.read_csv('data.csv')
  • From an Excel file:
Python
df = pd.read_excel('data.xlsx')

3. Basic DataFrame Operations:

  • Viewing data:
Python
df.head()       # First 5 rows
df.tail()       # Last 5 rows
df.info()       # DataFrame info
df.describe()   # Summary statistics
df.shape        # (rows, columns)
df.columns      # Column names
df.index        # Index values
  • Selecting data:
Python
df['col1']       # Select column 'col1'
df[['col1', 'col2']] # Select multiple columns
df.loc[0]         # Select row by label (index)
df.iloc[0]        # Select row by integer position
df[df['col1'] > 1] # Boolean indexing (filtering)
  • Adding/removing columns:
Python
df['new_col'] = [4, 5, 6] # Add a new column
df.drop('col1', axis=1)    # Remove column 'col1'
  • Adding/removing rows:
Python
df = df.append({'col1':4, 'col2':'d'}, ignore_index=True) #add row.
df.drop(0) #remove row by index.

4. Data Manipulation:

  • Sorting:
Python
df.sort_values(by='col1')
  • Grouping:
Python
df.groupby('col1').mean()
  • Applying functions:
Python
df['col1'].apply(lambda x: x * 2)
  • Handling missing values:
Python
df.isnull()       # Check for missing values
df.dropna()       # Remove rows with missing values
df.fillna(0)      # Fill missing values with 0
  • String operations (for string columns):
Python
df['col2'].str.upper() #convert to upper case.
df['col2'].str.contains('a') #boolean series if string contains a.
  • Merging/Joining:
Python
pd.merge(df1, df2, on='common_col') # Merge DataFrames
pd.concat([df1,df2]) #combine dataframes vertically
df1.join(df2, on='index', how='left') #join dataframes.

5. Time Series (if applicable):

  • Datetime conversion:
Python
df['date'] = pd.to_datetime(df['date'])
  • Resampling:
Python
df.resample('M', on='date').mean() #resample to monthly data.

6. Saving Data:

  • To CSV:
Python
df.to_csv('output.csv', index=False)
  • To Excel:
Python
df.to_excel('output.xlsx', index=False)

Important Notes:

  • axis=0 refers to rows, and axis=1 refers to columns.
  • inplace=True modifies the DataFrame directly, without creating a copy.
  • Always check the pandas documentation for the most up-to-date information.
Labels:

Comments

Post a Comment

Popular posts from this blog

Entity Relationship (ER) Diagram Model with DBMS Example

Reference :   Entity Relationship (ER) Diagram Model with DBMS Example What is ER Diagram? ER Diagram  stands for Entity Relationship Diagram, also known as ERD is a diagram that displays the relationship of entity sets stored in a database. In other words, ER diagrams help to explain the logical structure of databases. ER diagrams are created based on three basic concepts: entities, attributes and relationships. ER Diagrams contain different symbols that use rectangles to represent entities, ovals to define attributes and diamond shapes to represent relationships. At first look, an ER diagram looks very similar to the flowchart. However, ER Diagram includes many specialized symbols, and its meanings make this model unique. The purpose of ER Diagram is to represent the entity framework infrastructure. Entity Relationship Diagram Example Table of Content: What is ER Diagram? What is ER Model? History of ER models Why use ER Diagrams? Facts about ER Diagram Model ER Diagram...
Post a Comment
Read more

GCP Buckets

  CREATING BUCKET Method 1: Using GCP Console (UI) Go to the  Google Cloud Console . Navigate to  Cloud Storage . Click  Create Bucket . Enter a  unique name  for your bucket. Select a  storage class  (Standard, Nearline, Coldline, or Archive)... Choose a  location  (region/multi-region). Set  access control  (Uniform or Fine-grained). Click  Create . how do you optimise the class storage buckets - we can talk about cloud storage choosing wisely standard vs nearline vs coldline vs archive... how you managed your buckets - how did you manage? life cycle rule - regions multi regions etc
Post a Comment
Read more

Pyspark

pyspark link the above link for a sample code for Pyspark  Transformations create new datasets, actions return values, or write data. Transformations: Definition: Transformations are operations that create a new RDD or data frame from an existing one. They are "lazy," meaning they don't execute immediately. Instead, Spark builds a lineage graph (DAG - Directed Acyclic Graph) of transformations. Examples: map() : Applies a function to each element. filter() : Select elements based on a condition. flatMap() : Similar to a map, but flattens the results. groupByKey() : Groups elements by key. reduceByKey() : Reduces elements by key. join() : Joins two datasets. distinct() : Removes duplicate elements. union() : Combines two datasets. coalesce() : Reduces the number of partitions. repartition() : Increases or decreases the number of partitions. Actions: Definition: Actions trigger the execution of the lineage graph, which returns a result to the driver progra...
Post a Comment
Read more