Understanding Inferential Statistics and YARN in Python

Posted on Apr 2, 2025 in Computers

Inferential Statistics: Definition and Application in Python

Inferential Statistics involves making conclusions about a population based on a sample. It includes testing hypotheses, estimating population parameters, and predicting outcomes. Key concepts are:

  • Hypothesis Testing: Assessing evidence to support a hypothesis.
  • Confidence Intervals: Estimating a range for population parameters.
  • Regression Analysis: Predicting a dependent variable.
  • ANOVA: Comparing means across multiple groups.

Example: Hypothesis Testing Using a T-Test in Python

We will test if the mean test scores of two student groups (one using a study technique and one not) differ significantly.

Hypotheses:

  • Null Hypothesis (H₀): The mean scores of both groups are equal.
  • Alternative Hypothesis (H₁): The mean scores are different.

Steps:

  1. Collect sample data from both groups.
  2. Perform an independent t-test.
  3. Compare the p-value with the significance level (α = 0.05).

Python Code:

import numpy as np
from scipy import stats

# Test scores of two groups
group_A = [85, 90, 92, 87, 93, 88, 84, 91, 89, 90]
group_B = [78, 82, 80, 76, 79, 81, 77, 75, 80, 82]



# Perform t-test
t_stat, p_value = stats.ttest_ind(group_A, group_B)

# Interpretation
if p_value < 0.05:
    print("Reject H₀: Significant difference between
 the groups.")
else:
    print("Fail to reject H₀: No significant 
difference between the groups.")

Output:

Reject H₀: Significant difference between the groups.

Conclusion:

Since the p-value is less than 0.05, we reject the null hypothesis and conclude that the new study technique had a significant effect on the students’ test scores.

Application of Inferential Statistics: This technique helps in data-driven decision-making. It is used in various fields to estimate parameters, test treatments, and predict future outcomes. Inferential statistics is essential when analyzing sample data to make generalizations about a larger population. Python’s SciPy library provides convenient tools for performing these analyses.

Q. How can you read and manipulate CSV files using Python? Write a sample program.

To read and manipulate CSV files in Python, you can use either the csv module (built-in) or the pandas library. Here’s a concise explanation of both methods:

1. Using the csv Module

The csv module allows you to read and write CSV files by treating each row as a dictionary or a list. Here’s a simple example:


Code:

import csv

# Reading and printing CSV rows
with open('data.csv', mode='r') as file:
    csv_reader = csv.DictReader(file)
    for row in csv_reader:
        print(row)

# Manipulating data: Increase Age by 1
updated_rows = []
with open('data.csv', mode='r') as file:
    csv_reader = csv.DictReader(file)
    for row in csv_reader:
        row['Age'] = str(int(row['Age']) + 1)
        updated_rows.append(row)

# Writing updated data to a new file
with open('updated_data.csv', mode='w', newline='') as file:
    fieldnames = ['Name', 'Age', 'Department']
    csv_writer = csv.DictWriter(file, fieldnames=fieldnames)
    csv_writer.writeheader()
    csv_writer.writerows(updated_rows)

Explanation:

  • Reading: csv.DictReader() reads rows as dictionaries.
  • Manipulating: We increment the “Age” field by 1 for each row.
  • Writing: The updated data is written to a new file using csv.DictWriter().

2. Using the pandas Library

pandas provides a higher-level approach to handling CSV files and data manipulation using DataFrames.

Code:

import pandas as pd

# Reading the CSV file into a DataFrame
df = pd.read_csv('data.csv')

# Manipulating data: Increase Age by 1
df['Age'] = df['Age'] + 1

# Writing updated data to a new file
df.to_csv('updated_data_pandas.csv', index=False)

Explanation:

  • Reading: pd.read_csv() loads the CSV into a DataFrame.
  • Manipulating: We modify the “Age” column by adding 1 to each value.
  • Writing: The updated DataFrame is written to a new CSV file using to_csv().

Comparison:

  • csv Module: Lower-level, gives more control over reading/writing files manually. Suitable for small tasks.
  • pandas: High-level, more powerful for larger datasets and complex manipulations.

Conclusion: Use the csv module for simple tasks and pandas for more complex data manipulations and larger datasets.

Q. Role of YARN in Hadoop and Resource Management in a Distributed Environment

YARN (Yet Another Resource Negotiator) is the resource management layer of the Hadoop ecosystem, introduced in Hadoop 2.0 to address scalability and resource management challenges. It is responsible for managing resources in a distributed environment and for scheduling and monitoring jobs running on Hadoop clusters.

Key Functions of YARN:


  1. Resource Management: YARN manages resources across all the nodes in the cluster. It allocates resources to different applications and ensures efficient utilization.
  2. Job Scheduling: It schedules jobs based on resource availability and job priorities. YARN’s scheduler assigns tasks (such as Map and Reduce tasks) to available nodes in the cluster.
  3. Job Monitoring: YARN monitors the progress of jobs, including tracking their status and resource usage. It can also restart tasks if they fail.

How YARN Manages Resources:

  1. ResourceManager (RM): It is the master daemon responsible for managing cluster resources. It has two components:

    • Scheduler: Allocates resources based on policies (e.g., capacity, fairness).
    • ApplicationManager: Manages the lifecycle of applications running in the cluster.

  2. NodeManager (NM): It runs on each node and manages local resources, monitors resource usage (CPU, memory), and reports to the ResourceManager.

  3. Containers: YARN runs tasks in isolated environments called containers. A container is allocated a fixed amount of resources (memory, CPU) on a node and executes the task.

  4. ApplicationMaster (AM): Each application (e.g., a MapReduce job or Spark job) has its own ApplicationMaster, which negotiates with ResourceManager for resources, manages execution, and monitors job progress.

Resource Management Process:

  • When a job is submitted, the ResourceManager decides how much memory and CPU resources are required.


  • The ResourceManager allocates these resources to a node via NodeManager, which starts a container to run the task.
  • The ApplicationMaster negotiates with the ResourceManager to request resources for each phase of the job.
  • As tasks progress, NodeManagers track resource utilization and report it back to ResourceManager.

Conclusion:

YARN improves the scalability, flexibility, and efficiency of Hadoop by decoupling resource management from the MapReduce programming model. It supports multiple frameworks (MapReduce, Spark, etc.) to run concurrently in a shared cluster, ensuring optimal resource allocation and management in a distributed environment.