Onboarding a New Dataset

Purpose

This procedure defines the process for registering a new dataset within your data operating model. Following it ensures every dataset is properly catalogued, classified, and assigned clear ownership before it goes live.

Scope

This applies to all new datasets, regardless of source system or trust tier. It covers both internal sources (operational systems, ERP, CRM) and external feeds (IoT sensors, third-party APIs, partner data shares).

It does not cover changes to existing datasets (that is "Managing Schema Changes"), access requests for already-registered datasets ("Handling Data Access Requests"), or retiring datasets.

Prerequisites

• The source system is documented in the Platform Capabilities register
• A Data Owner (the business leader responsible for the dataset) has been identified and has accepted ownership
• Initial data profiling has been completed by the technical team
• Network connectivity to the source system is confirmed
• You have access to the Dataset Registry

Procedure

Step 1: Submit the registration request

Role: Data Steward (responsible for day-to-day upkeep within their domain)

Action: Create a new entry in the Dataset Registry with all required fields.

Details: Dataset name (descriptive, lowercase, underscores: customer_purchase_events, not CPE_data_v2); source system; a plain-English description of what the data contains and why it matters; schema definition; expected volume; update frequency; business purpose.

Expected outcome: A draft entry exists in the registry with all required fields populated.

Step 2: Classify the dataset

Role: Data Steward

Action: Apply your classification framework to determine the trust tier.

Details: Use the Classification Matrix to weigh four dimensions: data quality; business criticality (how many downstream processes depend on it); transformation level (raw as ingested, cleaned and conformed, or business-ready); and regulated content (personal data, financial data, anything else you are regulated on). The tier determines quality standards, access controls and monitoring.

Screening for personal data. "Personal data" is the term UK GDPR Article 4(1) uses: any information relating to an identified or identifiable living individual, directly or indirectly. Use it rather than the narrower American term "PII". Device identifiers, session IDs, IP addresses, cookie IDs and pseudonymised customer keys are all personal data, and none of them looks like a name. A schema scanned for first_name and email will pass a clickstream table that identifies every visitor. So read the column descriptions and a sample of the values, not just the names, and ask the source owner what a key joins to.

Then ask the narrower question: is any of it special category data under Article 9(1) (racial or ethnic origin, political opinions, religious or philosophical beliefs, trade union membership, genetic or biometric data, health, sex life or sexual orientation)? Criminal offence data sits under its own regime; treat it the same way here. Record the answer to both questions even when it is "no". A recorded "no" tells the next reviewer you asked. A blank tells them nothing.

Expected outcome: The dataset has a recorded trust tier, and the entry states whether it contains personal data and whether any of that is special category or criminal offence data.

Step 3: Assign ownership and access

Role: Data Owner (the business leader accountable for the dataset's quality and use)

Action: Confirm ownership and define initial access based on the tier and business need.

Details: Gold datasets require explicit Data Owner approval for each user or team. Silver datasets use team-level access. Bronze datasets are open to all authenticated users in the organisation.

Important: Datasets containing personal data require Data Protection Officer review regardless of tier. The tier tells you how refined the data is; it tells you nothing about who may lawfully see it, and a Bronze table full of raw event data is often the one carrying the most identifiers. Special category data raises the bar procedurally: processing it needs an Article 9(2) condition in addition to an ordinary lawful basis, and most UK conditions also require an appropriate policy document. Do not configure access to special category data while nobody can name the condition; hold it with access restricted to the onboarding team and escalate.

If your organisation has no Data Protection Officer, the Governance Lead handles this review. Most organisations of 50–1,000 people are not required to appoint one, so this is the normal case. Somebody named has to do the review, and it must not be the person registering the dataset.

Expected outcome: Access is configured and the Data Owner has formally accepted responsibility.

Step 4: Configure the ingestion pipeline

Role: Technical Custodian (the person who manages the data infrastructure)

Action: Set up the pipeline to your platform's ingestion standard, with error handling and retry logic, monitoring and alerting to the Custodian, and the pipeline documented in the Platform Capabilities register.

On Fabric: Data Factory, a Copy Activity with appropriate source and sink, triggers matching the agreed frequency. On AWS: Glue Jobs or Step Functions, a Glue Crawler for schema discovery where appropriate, CloudWatch alarms on failure.

Expected outcome: A working pipeline on the agreed schedule, with error handling and monitoring in place.

Step 5: Validate and test

Role: Technical Custodian

Action: Run the quality checks for the dataset's tier and record the results.

Flag any issues before promotion. Bronze datasets may accept lower thresholds, but the checks must still be run and documented.

Expected outcome: All checks recorded; any failures documented with a remediation plan.

Step 6: Publish to the registry

Role: Data Steward

Action: Once all checks pass, set the entry from Draft to Active, notify the Data Owner and the Business Consumers who will use it, and record the approval.

Expected outcome: The dataset is live in the registry, discoverable, and fully owned.

Quick reference checklist

• Registration request submitted with all required metadata
• Trust tier assigned using the Classification Matrix
• Data Owner has accepted ownership
• Entry records whether the dataset contains personal data, and whether any is special category or criminal offence data
• Access configured, with DPO or Governance Lead review completed if it contains personal data
• Ingestion pipeline configured, documented and tested
• Quality checks executed and recorded
• Status set to Active
• Data Owner and Business Consumers notified

Approval

The other six

Handling Data Access Requests. Managing Schema Changes. Data Tier Promotion. Retiring a Dataset. Incident Handling. Quality Reviews. Each is written to this standard, with a named role and an expected outcome for every step, and each is yours to edit. How the procedures work inside the product.