File Handling in Python

In this lab, you will learn how to handle files in Python. File handling allows you to read from and write to files, which is essential for data processing and storage. You can perform the exercises in any of the following environments:

• Visual Studio Code (VS Code)
• Google Colab
• Jupyter Notebook
• PyCharm
• Anaconda

Setup Instructions

1. Visual Studio Code (VS Code)

• Installation: Ensure Python is installed and the Python extension is added to VS Code.
• Create a New File: Open VS Code and create a new Python file (e.g., file_handling_lab.py).
• Run the Code: Use the terminal or the run button in the editor to execute your code.

2. Google Colab

• Access Colab: Go to Google Colab.
• Create a New Notebook: Click on “New Notebook.”
• Run Cells: Write your code in a cell and run it by clicking the play button.

3. Jupyter Notebook

• Installation: Install Jupyter Notebook via Anaconda or pip.
• Start Jupyter: Open your terminal and type jupyter notebook to launch.
• Create a New Notebook: Create a new Python notebook and write your code in the cells.

4. PyCharm

• Installation: Download and install PyCharm.
• Create a New Project: Start a new project and create a new Python file.
• Run the Code: Write your code and run it using the built-in run configuration.

5. Anaconda

• Launch Anaconda Navigator: Open Anaconda Navigator.
• Choose an Environment: Launch either Jupyter Notebook or Spyder.
• Create a New File/Notebook: Write your code in a new notebook or script.

Exercises

Exercise 1: Writing to a File

Task:

1. Create a text file and write some text into it.

Example Code:

# Write to a file
with open('example.txt', 'w') as file:
    file.write("Hello, this is a sample text file.\n")
    file.write("File handling in Python is easy and efficient.")

Exercise 2: Reading from a File

Task:

1. Read the contents of the file you just created and print it.

Example Code:

# Read from the file
with open('example.txt', 'r') as file:
    content = file.read()
    print("File Content:\n", content)

Exercise 3: Appending to a File

Task:

1. Append more text to the existing file.

Example Code:

# Append to the file
with open('example.txt', 'a') as file:
    file.write("\nThis line is appended to the file.")

Exercise 4: Reading File Line by Line

Task:

1. Read the file line by line and print each line.

Example Code:

# Read the file line by line
with open('example.txt', 'r') as file:
    for line in file:
        print(line.strip())  # Use strip() to remove leading/trailing whitespace

Exercise 5: Handling Exceptions

Task:

1. Handle exceptions when trying to read a non-existent file.

Example Code:

# Handle exceptions
try:
    with open('non_existent_file.txt', 'r') as file:
        content = file.read()
except FileNotFoundError:
    print("Error: The file does not exist.")

Exercise 6: Using JSON for File Handling

Task:

1. Write and read a JSON file.

Example Code:

import json

# Data to be written to JSON file
data = {
    "name": "John",
    "age": 30,
    "city": "New York"
}

# Write to JSON file
with open('data.json', 'w') as json_file:
    json.dump(data, json_file)

# Read from JSON file
with open('data.json', 'r') as json_file:
    loaded_data = json.load(json_file)
    print("Loaded Data:", loaded_data)

Running the Code

1. Copy the Code: Copy the code examples provided above into your Python file.
2. Run the Program: Execute the program in your IDE. You can usually do this by clicking a “Run” button or using a keyboard shortcut (like Shift + F10 in PyCharm).
3. Check the Files: After running the code, check the directory where your script is located to see the created files (example.txt and data.json).

Conclusion

This lab helps you understand how to handle files in Python, including writing to, reading from, appending to, and handling exceptions for files. You also learned how to work with JSON files, which are commonly used for data interchange. By completing these exercises, you will gain practical experience in file handling in Python.

File Handling in Python for Civil Engineering: Advanced Practices & Applications

File handling is essential in civil engineering for managing diverse data types, from sensor readings to geospatial data. Below is an enhanced guide focusing on scenario-specific workflows, advanced libraries, and practical automation tailored to civil engineering tasks.

Scenario-Based File Handling Workflows

1. Sensor Data Analysis (CSV/HDF5)

• Use Case: Process real-time bridge vibration data from IoT sensors.
• Libraries: pandas (CSV), h5py (HDF5), Dask (large datasets).
• Example: Aggregate and filter terabytes of vibration data.
  python import dask.dataframe as dd # Read large CSV in chunks ddf = dd.read_csv("sensor_data_*.csv", blocksize=1e6) # 1MB chunks # Parallel computation for peak acceleration max_accel = ddf['acceleration'].max().compute()

2. Geospatial Data Processing (Shapefiles/GeoJSON)

• Use Case: Analyze flood risk zones using GIS data.
• Libraries: geopandas, rasterio (for elevation rasters).
• Example: Calculate flood-prone areas from elevation data.
  python import geopandas as gpd elevation = gpd.read_file("terrain_elevation.shp") flood_zones = elevation[elevation['elevation'] < 10] # Areas below 10m flood_zones.to_file("flood_risk_zones.geojson", driver='GeoJSON')

3. Material Test Databases (SQLite/Parquet)

• Use Case: Store and query concrete strength test results.
• Libraries: sqlite3, pyarrow (Parquet).
• Example: Use SQLite for local database management.
  python import sqlite3 conn = sqlite3.connect('concrete_tests.db') # Query mixes with strength > 30 MPa df = pd.read_sql("SELECT * FROM mixes WHERE strength_MPa > 30", conn)

4. Automated Report Generation (Excel/PDF)

• Use Case: Generate design reports with tables and plots.
• Libraries: openpyxl, reportlab (PDF), matplotlib.
• Example: Create a PDF report with analysis results. from reportlab.lib.pagesizes import A4 from reportlab.pdfgen import canvas c = canvas.Canvas("structural_report.pdf", pagesize=A4) c.drawString(100, 750, "Peak Stress: 45.2 MPa (Safety Factor = 2.1)") c.save()

Advanced Libraries & Techniques

1. Cloud Integration

• Use Case: Access LiDAR data stored in AWS S3.
• Library: boto3 (AWS), adlfs (Azure).
• Example: Download terrain data from S3.
  python import boto3 s3 = boto3.client('s3') s3.download_file('my-bucket', 'terrain_data.las', 'local_terrain.las')

1. Binary Files for Simulation Outputs

• Use Case: Store finite element analysis (FEA) results efficiently.
• Library: pickle, numpy (.npy format).
• Example: Save/load stress tensors.
  python np.save("fea_stresses.npy", stress_tensor) # Save stress_tensor = np.load("fea_stresses.npy") # Load

1. Version Control for Configurations

• Use Case: Track changes in soil parameters for slope stability models.
• Workflow: Store JSON configurations in Git.
  python import json params = { "cohesion_kPa": 25, "friction_angle_deg": 30 } with open("slope_params.json", "w") as f: json.dump(params, f) # Commit to Git for versioning

Robust Data Validation & Error Handling

1. Unit Consistency Checks

   def validate_stress_data(df):
       if df['stress_MPa'].max() > 500:  # Unrealistic for concrete
           raise ValueError("Stress exceeds material limits (500 MPa).")

2. Geospatial Data Sanity Checks

   roads = gpd.read_file("roads.shp")
   if not roads.crs == 'EPSG:4326':  # Ensure WGS84 coordinate system
       roads = roads.to_crs('EPSG:4326')

Case Study: Construction Site Monitoring

Task: Analyze temperature and strain data from embedded sensors to detect anomalies.

import pandas as pd
from sklearn.ensemble import IsolationForest

# 1. Load sensor data
sensors = pd.read_csv("site_sensors.csv", parse_dates=['timestamp'])

# 2. Detect anomalies using machine learning
model = IsolationForest(contamination=0.01)
sensors['anomaly'] = model.fit_predict(sensors[['temp_C', 'strain']])

# 3. Save flagged anomalies to Excel
anomalies = sensors[sensors['anomaly'] == -1]
with pd.ExcelWriter("anomalies.xlsx") as writer:
    anomalies.to_excel(writer, sheet_name="High Risk")

# 4. Archive raw data to HDF5
sensors.to_hdf("sensor_archive.h5", key="2023-10-01", mode='a')

Best Practices for Civil Engineering Workflows

1. Automate Repetitive Tasks

• Use cron (Linux) or Task Scheduler (Windows) to run nightly data processing scripts.

1. Data Privacy

• Encrypt sensitive project files (e.g., client reports) with libraries like cryptography.

1. Modular Code Design

• Organize file handling into reusable functions:
  python def load_sensor_data(path): df = pd.read_csv(path) df['timestamp'] = pd.to_datetime(df['timestamp']) return df

1. 3D Model Handling

• Use trimesh or pyvista to read/write CAD files (e.g., STL, OBJ) for structural components.

Key Tools & Libraries

Conclusion

Python’s versatility in file handling empowers civil engineers to tackle domain-specific challenges, from real-time sensor analytics to geospatial modeling. By integrating advanced libraries (e.g., Dask for big data), cloud tools, and robust validation checks, engineers can ensure efficient, secure, and scalable data workflows. Always align file formats with project needs—use HDF5 for simulations, Parquet for queryable databases, and GeoJSON for geospatial clarity.