Module 4 Lab Companion

↑ Back to Lab Contents

Module 4 Learning Goals

This lab is light on new code. It is about professional habits: a notebook someone else can rerun, names a reader understands, documentation of what each column means, and careful decisions about who may see which data. By the end of the lab you should be able to:

• Use reusable, relative file paths with pathlib.Path.
• Write readable Python with descriptive snake_case names.
• Organize a notebook so its purpose, inputs, checks, decisions and outputs are visible.
• Draft, improve and save a data dictionary.
• Tell apart direct identifiers, confidential identifiers, quasi-identifiers and sensitive business fields.
• Create and verify a review copy for one specific audience, and explain responsible AI and tool use.

The main idea: someone else should be able to rerun your work, understand it, and know how the data may be shared.

↑ Back to Lab Contents

Before You Start: Repository and Setup

Module 4 uses the same repository you created for Module 3 from the template tianhaiz/ban6003-week-03-04-cleaning-reproducibility-template. Do not create a new one, and do not work in the instructor template.

1. Open your own Week 3–4 repository in GitHub Codespaces (main path). If you create a new Codespace, the first setup takes about 4–5 minutes.
2. Open notebooks/week4_github_repro_governance_lab.ipynb. Select the kernel if VS Code asks.
3. Run the cells from the top. If you restart the kernel, run from the top again.
4. Keep the repository public, or invite the instructor account zzz1990771 as a collaborator.

Optional local path: open the repository with GitHub Desktop (Open in Visual Studio Code), trust the folder when VS Code asks, and choose the base conda environment as the kernel.

↑ Back to Lab Contents

Part 1: Imports and Reusable Paths

A reproducible notebook shows clearly where its data comes from and where its outputs go. Import packages first, at the top, because every later cell uses them. Then define the paths once, near the top.

1.1 Imports, paths and an input check

from pathlib import Path
import pandas as pd

# Project-style paths
RAW_DATA_PATH = Path("../data/raw/employee_data.csv")
PROCESSED_DATA_DIR = Path("../data/processed")
PROCESSED_DATA_DIR.mkdir(parents=True, exist_ok=True)

if not RAW_DATA_PATH.is_file():
    raise FileNotFoundError(f"Required raw input was not found: {RAW_DATA_PATH}")

print("Raw data path:", RAW_DATA_PATH)
print("Processed data folder:", PROCESSED_DATA_DIR)

↑ Back to Lab Contents

Part 2: Load the Data with a Descriptive Name

employees_raw = pd.read_csv(RAW_DATA_PATH)
employees_raw.head()

This is the same raw employee file as in Module 3, so the problems will look familiar: a mixed date format, not a date, thirty as an age, $ and commas in salaries, and Yes/Y in full_time.

Check the column types

employees_raw.info()

↑ Back to Lab Contents

Part 3: Clean Code Style in Practice

Readable code is not complicated code. Four habits help: snake_case names, descriptive names, important paths and settings near the top, and comments that explain why rather than repeat what the code says.

3.1 Profile values with clear names

row_count, column_count = employees_raw.shape
salary_missing_count = employees_raw["salary"].isna().sum()
duplicate_employee_id_count = employees_raw["employee_id"].duplicated().sum()

print(f"Rows: {row_count}")
print(f"Columns: {column_count}")
print(f"Missing salary values: {salary_missing_count}")
print(f"Duplicate employee_id values: {duplicate_employee_id_count}")

↑ Back to Lab Contents

Your Turn: Improve the Names

# Original version
x = employees_raw["age"].isna().sum()
y = employees_raw["department"].value_counts()

print(x)
print(y)

age_missing_count = employees_raw["age"].isna().sum()
department_value_count = employees_raw["department"].value_counts()

print(age_missing_count)
print(department_value_count)

1
department
Finance      3
Sales        2
HR           2
IT           2
Marketing    2
sales        1
Name: count, dtype: int64

↑ Back to Lab Contents

Part 4: Notebook Organization

A professional notebook tells a story, not just code and results. A useful structure is: purpose → inputs → data checks → decisions or observations → outputs. Use it for the whole notebook and also for each small task: before a code block, say what you are checking; after it, say what you found.

4.1 Count missing values in every column

employees_raw.isna().sum()

↑ Back to Lab Contents

Part 5: Build a Practical Data Dictionary

A data dictionary is the most common kind of metadata (data about data). It tells another person what each column means and how it must be handled. This one has six columns: column_name, current_dtype, business_description, sensitivity_level, cleaning_or_governance_notes and sharing_guidance. The last one turns a classification into an action: who may use the field, or what must happen before it is shared.

5.1 Create the starter dictionary

data_dictionary = pd.DataFrame({
    "column_name": employees_raw.columns,
    "current_dtype": [str(dtype) for dtype in employees_raw.dtypes],
    "business_description": [
        "Internal employee identifier",
        "Employee first name",
        "Employee last name",
        "Employee hire date",
        "Employee department",
        "Employee salary",
        "Employee age",
        "Whether the employee works full time"
    ],
    "sensitivity_level": [
        "Confidential internal identifier",
        "Direct identifier / PII",
        "Direct identifier / PII",
        "Quasi-identifier",
        "Business attribute",
        "Sensitive financial/business information",
        "Quasi-identifier",
        "Employment attribute"
    ],
    "cleaning_or_governance_notes": [
        "Should be unique; validate non-numeric and duplicate values",
        "Not needed for most analytical summaries",
        "Not needed for most analytical summaries",
        "Parse and validate dates before analysis",
        "Clarify valid categories and business definitions",
        "Convert to numeric and investigate missing or invalid values",
        "Convert to numeric and investigate impossible values",
        "Clarify meaning and standardize valid values"
    ],
    "sharing_guidance": [
        "Restrict to authorized users who need record-level traceability",
        "Remove unless the approved purpose requires a name",
        "Remove unless the approved purpose requires a name",
        "Generalize or remove for broad sharing",
        "Review disclosure risk when combined with other fields",
        "Restrict access; aggregate for broad audiences",
        "Group into ranges or remove for broad sharing",
        "Use only when relevant to the approved purpose"
    ]
})

data_dictionary

↑ Back to Lab Contents

Your Turn: Improve the Data Dictionary

data_dictionary.loc[
    data_dictionary["column_name"] == "salary",
    "cleaning_or_governance_notes",
] = (
    "Stored as text: remove $ and commas, then convert to numeric. "
    "One value is missing and one says 'invalid'; confirm both with the HR system owner."
)
data_dictionary.loc[
    data_dictionary["column_name"] == "full_time",
    "business_description",
] = "Whether the employee works full time; recorded as Yes/Y (full time) or No/N (not full time)"

data_dictionary.loc[
    data_dictionary["column_name"].isin(["salary", "full_time"]),
    ["column_name", "business_description", "cleaning_or_governance_notes"],
]

5.2 Save the dictionary

dictionary_output_path = PROCESSED_DATA_DIR / "employee_data_dictionary.csv"
data_dictionary.to_csv(dictionary_output_path, index=False)
print("Saved data dictionary to:", dictionary_output_path)

↑ Back to Lab Contents

Part 6: Identify PII and Sensitive Fields

PII (personally identifiable information) is information that identifies a person. Some columns identify a person directly. Others become identifying when combined. Some do not identify anyone but still carry business or ethical risk.

6.1 Sensitivity summary

sensitivity_summary = data_dictionary[["column_name", "sensitivity_level"]]
sensitivity_summary

↑ Back to Lab Contents

Part 7: Create a Limited Internal Review Copy

Scenario: an authorized HR data-quality reviewer needs row-level records and employee IDs to trace source-system problems, but does not need names. You create a copy without first_name and last_name. This file is not anonymous: it still has a confidential ID, exact dates, age and salary, so it stays with the approved internal audience.

7.1 Drop the name columns and verify

direct_name_columns = ["first_name", "last_name"]
employees_review_copy = employees_raw.drop(columns=direct_name_columns).copy()

# Verify that names were removed without changing the number of records.
assert len(employees_review_copy) == len(employees_raw)
assert set(direct_name_columns).isdisjoint(employees_review_copy.columns)
assert set(direct_name_columns).issubset(employees_raw.columns)

employees_review_copy.head()

7.2 Save the review copy

review_output_path = PROCESSED_DATA_DIR / "employee_review_copy_no_names.csv"
employees_review_copy.to_csv(review_output_path, index=False)
print("Saved restricted internal review copy to:", review_output_path)

7.3 Verify the saved file, not only the DataFrame

saved_review_copy = pd.read_csv(review_output_path)

guided_release_checks = pd.Series({
    "source_rows": len(employees_raw),
    "saved_review_rows": len(saved_review_copy),
    "rows_match": len(saved_review_copy) == len(employees_raw),
    "first_name_present": "first_name" in saved_review_copy.columns,
    "last_name_present": "last_name" in saved_review_copy.columns,
    "employee_id_present": "employee_id" in saved_review_copy.columns,
    "salary_present": "salary" in saved_review_copy.columns,
})

guided_release_checks

↑ Back to Lab Contents

Part 8: Responsible AI and Tool-Use Checkpoint

↑ Back to Lab Contents

Part 9: GitHub Workflow Checkpoint

Save the notebook first. Then review what will be committed with git status, and use a commit message that says what the work did.

git status
git add .
git commit -m "Complete Week 4 documentation and governance lab"
git push

Run these in the terminal (Terminal → New Terminal), not in a notebook cell. In Codespaces you can also use the Source Control panel; with the local path, use GitHub Desktop: commit, then Push origin.

↑ Back to Lab Contents

Part 10: Offline Assignment: Release-Readiness Audit

# Offline Assignment
# Hint: use `"column_name" in offline_review.columns` for each remaining-field check.
# The direct-name check is already completed as an example.
offline_review = pd.read_csv(review_output_path)

release_readiness_summary = pd.DataFrame([{
    "source_rows": len(employees_raw),
    "review_rows": len(offline_review),
    "rows_match": len(employees_raw) == len(offline_review),
    "direct_name_columns_present": bool(set(direct_name_columns) & set(offline_review.columns)),
    "employee_id_present": None,  # TODO
    "salary_present": None,  # TODO
    "exact_age_present": None,  # TODO
    "exact_hire_date_present": None,  # TODO
}])

release_readiness_summary

↑ Back to Lab Contents

Optional Practice: A Purpose-Limited Copy (Not Graded)

This practice does not count toward your grade. Scenario: an approved internal manager needs only the recorded department and full_time categories. Keep only those two columns, save, reload and check.

approved_columns = ["department", "full_time"]
restricted_columns = {"employee_id", "first_name", "last_name", "hire_date", "salary", "age"}

manager_category_copy = employees_raw[approved_columns].copy()
manager_copy_path = PROCESSED_DATA_DIR / "optional_manager_category_copy.csv"
manager_category_copy.to_csv(manager_copy_path, index=False)

saved_manager_category_copy = pd.read_csv(manager_copy_path)
optional_release_checks = pd.Series({
    "rows_match_source": len(saved_manager_category_copy) == len(employees_raw),
    "only_approved_columns": set(saved_manager_category_copy.columns) == set(approved_columns),
    "restricted_columns_present": sorted(restricted_columns & set(saved_manager_category_copy.columns)),
})

optional_release_checks

rows_match_source             True
only_approved_columns         True
restricted_columns_present      []
dtype: object

# Run this self-check after you create saved_manager_category_copy and optional_release_checks.
required_practice_objects = {
    "saved_manager_category_copy",
    "optional_release_checks",
}

if not required_practice_objects.issubset(globals()):
    print("Complete the optional practice first, then rerun this self-check.")
else:
    expected_columns = {"department", "full_time"}
    restricted_columns = {"employee_id", "first_name", "last_name", "hire_date", "salary", "age"}
    print("Rows match source:", len(saved_manager_category_copy) == len(employees_raw))
    print("Only approved columns remain:", set(saved_manager_category_copy.columns) == expected_columns)
    print("Restricted columns present:", sorted(restricted_columns & set(saved_manager_category_copy.columns)))
    display(optional_release_checks)

↑ Back to Lab Contents

Common Problems

If a cell fails, read the last line of the error message first. Then find it below.

Still stuck? Post the error message and the cell number in the course Q&A discussion. Do not post your full answers.

↑ Back to Lab Contents

Save and Submit

1. Complete the name rewrite, the observation, the dictionary edits, the privacy reflection, the AI response, the offline summary and the release decision.
2. Restart the kernel and run all cells from top to bottom. Confirm there are no errors.
3. Confirm that both files exist in data/processed: employee_data_dictionary.csv and employee_review_copy_no_names.csv.
4. Save the notebook, check git status, then commit and push.
5. Open your repository on GitHub to confirm everything arrived, and submit the repository link in Canvas.

↑ Back to Lab Contents

Lab Wrap-Up

This lab added professional habits to the cleaning work of Module 3:

• Path, relative paths, mkdir(exist_ok=True) and is_file(): a notebook that runs on any computer.
• Descriptive snake_case names and a purpose → inputs → checks → decisions → outputs structure: work others can read.
• A saved data dictionary: metadata that tells people what each field means and how to share it.
• drop(...).copy(), assert and reloading the saved file: verified, audience-specific outputs.
• Responsible tool use: data ownership and policy first, and you remain accountable.

Removing names alone does not make a file anonymous. Who receives the data, and for what purpose, is part of every release decision. Module 5 starts transforming data with query(), sort_values() and assign(); keep using the naming and path habits from this lab.