2. Things data engineer should know

 

data enginner should know how to write and read python code to extract the data from the internal and external apps .

1. Need understand what is API call means .

if we search in the front end (GUI) the results will be fetched back  database through API call .

a. We need to know to write the python code to fetch the details through API calls .

its possible by using Request library in python.

2. Need to understand the use case of the company.

example : telecom company (news channel).

the company like news channels , weather report like shown in the diagram .

it can be acheive through  by exposing certain API by the company X to teh other company . another company gets the details by sending the API calls .

The structure of an API call is generally the same across industries. In telecom, APIs are commonly used to manage subscribers, retrieve usage, send SMS, provision services, or check network status.

General Structure of an API Call

HTTP Method + URL + Headers + Parameters/Body

1. HTTP Method

Defines what action you want to perform.

HTTP Method	Purpose	Telecom Example	Idempotent?
GET	Retrieve data	Get subscriber details or account balance	✅ Yes
POST	Create a new resource or perform an action	Activate a new SIM or recharge an account	❌ No
PUT	Update or replace an existing resource	Change a customer's tariff plan	✅ Yes
DELETE	Remove a resource	Deactivate a service or cancel a subscription	✅ Yes
PATCH	Partially update an existing resource	Update only the customer's email or roaming status	❌ No (typically)

---

2. Endpoint (URL)

The endpoint specifies the resource.

Example:

https://api.telecom.com/v1/subscribers/9876543210

Here:

• https:// → Protocol

• api.telecom.com → Server

• /v1 → API version

• /subscribers → Resource

• 9876543210 → Subscriber ID (or MSISDN)

---

3. Headers

Headers provide metadata about the request.

Example:

Authorization: Bearer eyJhbGciOi...
Content-Type: application/json
Accept: application/json

Common telecom headers include:

• Authorization Token

• API Key

• Transaction ID

• Correlation ID

• Content Type

---

4. Parameters

Parameters pass additional information.

Query Parameters

Example:

GET /usage?month=June&year=2026

Full URL:

https://api.telecom.com/v1/usage?month=June&year=2026

---

Path Parameters

GET /subscribers/9876543210

Here:

9876543210

is the subscriber identifier.

---

5. Request Body

Used mainly with POST or PUT requests.

Example: Activate a SIM

{
  "customerId": "C12345",
  "msisdn": "9876543210",
  "plan": "Premium_5G",
  "simSerial": "8991101200003204512"
}

---

Telecom Example 1: Get Subscriber Details

Request

GET /v1/subscribers/9876543210

Headers

Authorization: Bearer <token>
Accept: application/json

Response

{
  "subscriberId": "9876543210",
  "status": "Active",
  "plan": "Unlimited 5G",
  "balance": 120.50
}

---

Telecom Example 2: Activate a SIM

Request

POST /v1/sim/activate

Headers

Authorization: Bearer <token>
Content-Type: application/json

Body

{
  "customerId": "C12345",
  "simSerial": "8991101200003204512",
  "plan": "Basic"
}

Response

{
  "status": "Success",
  "activationId": "ACT123456",
  "message": "SIM activated successfully."
}

---

Telecom Example 3: Recharge Account

Request

POST /v1/recharge

Body

{
  "msisdn": "9876543210",
  "amount": 50
}

Response

{
  "transactionId": "TX987654",
  "status": "SUCCESS",
  "newBalance": 170.50
}

---

Telecom Example 4: Check Data Usage

Request

GET /v1/usage/9876543210?type=data

Response

{
  "subscriber": "9876543210",
  "usedGB": 12.4,
  "remainingGB": 17.6,
  "billingCycle": "June-2026"
}

---

End-to-End API Call Flow in Telecom

Client (Mobile App / CRM)
          │
          │ HTTP Request
          ▼
+----------------------------+
| API Gateway                |
| Authentication             |
| Rate Limiting              |
+----------------------------+
          │
          ▼
+----------------------------+
| Telecom Service API        |
| Subscriber Service         |
| Billing Service            |
| Network Service            |
+----------------------------+
          │
          ▼
+----------------------------+
| Telecom Backend Systems    |
| HLR/HSS                    |
| CRM                        |
| Billing                    |
| Provisioning               |
| Usage Database             |
+----------------------------+
          │
          │ Response
          ▼
Client receives JSON response

Complete Example

HTTP Request

POST https://api.telecom.com/v1/recharge
Authorization: Bearer abc123xyz
Content-Type: application/json

{
    "msisdn":"9876543210",
    "amount":100
}

Processing

1. The client sends a POST request to the recharge endpoint.

2. The API Gateway validates the authentication token.

3. The recharge service processes the request.

4. The billing system updates the subscriber's balance.

5. The service returns a response.

HTTP Response

HTTP/1.1 200 OK
Content-Type: application/json

{
    "transactionId":"TX1002456",
    "status":"SUCCESS",
    "newBalance":220.50
}

This example shows all the key components of an API call:

• Method: POST

• Endpoint: /v1/recharge

• Headers: Authorization, Content-Type

• Request Body: Recharge details (MSISDN and amount)

• Response: HTTP status code and a JSON payload containing the transaction result and updated balance.

Hands on :

free api : # https://restcountries.com/v3.1/all

import requests

# https://restcountries.com/v3.1/rc_live_b4d27ddee40740e5aca33f36f24c2358 -- api url

import requests

response = requests.get(

  'https://api.restcountries.com/countries/v5?q=canada',

  headers={'Authorization': 'Bearer rc_live_b4d27ddee40740e5aca33f36f24c2358'}

)

data = response.json()

print (data)

Here output is in JSON foramt --> dictionary --> key , value pairs.

its in semi structured format , means we can perdict what is inside partically not fully 

What is Semi-Structured Data?

Semi-structured data does not follow a fixed table structure like a relational database. However, it does have some organization using keys, tags, or attributes, so we can partially predict its structure.

For example, consider addresses from different countries.

India

{
    "country": "India",
    "address": {
        "street_name": "MG Road",
        "city": "Bengaluru",
        "pincode": "560001"
    }
}

United Kingdom

{
    "country": "United Kingdom",
    "address": {
        "street_name": "Baker Street",
        "city": "London",
        "postcode": "NW1 6XE"
    }
}

Why is this semi-structured?

We can predict that:

• There will be an address object.
• It will probably contain a street_name and city.

But we cannot always predict every field:

• India uses pincode.
• The UK uses postcode.
• Another country might use zip_code.

So the structure is similar but not identical.

Comparison

Structured Data	Semi-Structured Data
Fixed schema	Flexible schema
Every record has the same columns	Records can have different fields
Example: SQL tables	Example: JSON, XML
Easy to query	More flexible but may require extra handling

A simple definition for interviews

Semi-structured data is data that has an organized format using keys or tags, but it does not follow a fixed schema. Different records may contain different fields, although they share a similar overall structure. JSON and XML are common examples of semi-structured data.