Every organisation is different, so there is a range of data management methods available. Organisations are free to combine data management practices in a way that suits them, but the following techniques are among the most widely used:
Data pipelines
A data pipeline provides an automated pathway for transferring information between two or more systems. For example, you might connect your sales enablement software to your website analytics to enrich lead profiles. At times, the data pipeline may modify or enhance data during the exchange process, though it can equally leave the raw data unchanged.
Example: A retail company uses a data pipeline to automatically transfer sales data from its CRM system into a cloud-based data warehouse. This enables the company to generate real-time sales reports without manual data entry.
ETL and ELT are specific types of data pipeline used for data integration. ETL involves extracting data from source systems, transforming it into a suitable format, and loading it into a target system – often a data warehouse. ELT reverses the order of those final two steps, loading the raw data first and then transforming it within the target system.
Example: A healthcare provider extracts patient data from multiple clinics (extract), cleans and formats it to comply with privacy regulations (transform), then loads it into a central data warehouse for analysis (load).
For ELT, consider a social media platform that extracts user activity data (extract), loads it into a data lake (load), and then processes the data to generate user engagement insights (transform).
Data architecture
Data architecture defines the overall framework governing how data flows through an organisation’s systems, covering everything from storage and usage to compliance. A well-defined architecture ensures information is managed efficiently and consistently.
Example: A financial institution designs its data architecture to ensure customer transactions are stored securely and in compliance with industry regulations. Data is held in a secure data warehouse, with specific access control policies defined for each department.
Data modelling
Data modelling involves creating visual diagrams that represent the structure of data and the relationships between different data elements, whether within a single system or across multiple systems. Data models give teams a clear understanding of how data flows and is organised, supporting more effective management and analysis.
Example: A logistics company creates a data model to visualise the relationships between warehouses, inventory items, and shipping routes. This helps the company optimise its inventory management by offering a clearer understanding of how products move through the system.
Data catalogues
Data catalogues act as structured inventories of an organisation’s data assets, holding metadata that makes essential information searchable and readily discoverable. For instance, a data catalogue can store details about the location, format, and quality of various datasets.
Example: A large university maintains a data catalogue, enabling researchers to access datasets across various academic fields with ease. The catalogue includes metadata such as dataset descriptions, formats, and usage restrictions.
Data governance
Data governance encompasses the rules, policies, and procedures an organisation follows to standardise data, ensuring its quality, security, and compliance. Establishing a dedicated team to oversee data policies and maintain accountability is a common and effective approach.
Example: A pharmaceutical company puts data governance practices in place to ensure clinical trial data is accurate, consistent, and compliant with regulatory standards. A dedicated team oversees these practices, maintaining proper documentation and audit trails.
Data security
The primary goal of data security is to protect an organisation’s information from breaches, theft, and unauthorised access. This IT function typically involves creating and enforcing policies covering software, access controls, backups, and storage.
Example: An e-commerce company encrypts sensitive customer data (such as credit card numbers) and implements two-factor authentication for employees accessing the system, ensuring only authorised individuals can retrieve the data.
Data life cycle management
Data life cycle management involves monitoring and managing data throughout its entire life cycle – from creation or collection through to eventual deletion or archiving. Establishing clear policies for each stage ensures data is handled appropriately, remains relevant, and stays secure.
Example: A government agency introduces a policy to archive citizen data after 10 years. The aim is to keep active data readily accessible while minimising the storage costs associated with older, less-relevant records.
Data processing
At its core, data processing is the transformation of raw data into a usable and actionable format. The process typically encompasses cleaning, transforming, and integrating data to derive meaningful insights.
Example: A media company collects raw data from various video streaming platforms, processes it to remove irrelevant information, and structures it in a database to deliver personalised recommendations to viewers.
Data integration
Data integration brings together information from multiple disparate sources into a unified view. For organisations that depend on several systems across different operations, this approach delivers a comprehensive understanding of their data.
Example: An airline integrates data from its booking system, customer service platform, and social media channels. The goal is to build a unified, comprehensive view of each customer’s interactions and preferences – enhancing both customer service and overall marketing efforts.
Data migration
Data migration involves transferring data between different systems or platforms – typically when upgrading to a new database solution or moving data to the cloud. The objective is to transfer existing information to the new environment with minimal errors or formatting issues.
Example: A retail chain migrates its inventory data from an On-Premise database to a cloud-based system. This enables real-time tracking and greater scalability as the business grows.
Data storage
As a fundamental aspect of data management, this discipline involves securely saving data in a chosen location – whether on physical servers or in the cloud. Selecting the right storage solution depends on several factors, including data volume, access frequency, and security requirements.
Example: A media company stores video files on high-capacity Cloud storage, enabling straightforward scaling as content production grows. Data is regularly backed up to guard against loss.
Master data management (MDM)
Master data management centres on ensuring that core business data – such as customer or product information – is accurate, consistent, and shared across the organisation. This reduces duplication and errors, providing a single source of truth for critical data
Example: A global retailer uses MDM to maintain a single, consistent record of all products across their stores. This reduces errors in product listings and improves worldwide inventory management.
Big data management
As data volumes continue to grow, big data management techniques become essential for handling and analysing vast amounts of information from diverse sources – often including unstructured or semi-structured data. This typically involves technologies such as data lakes and specialised processing frameworks.
Example: A technology company uses big data management tools to analyse user behaviour across millions of devices. By processing data in a distributed manner, the company gains insights into user preferences and refines its product recommendations.
Cloud data management
As more organisations move their data to the cloud, cloud data management has become an increasingly critical discipline. This involves managing data within cloud-based environments, drawing on the cloud’s scalability, flexibility, and cost-effectiveness.
Example: A start-up uses cloud data management to store and process large volumes of customer data in real time. This enables the company to scale cloud computing resources during periods of peak demand and keep operational costs low during quieter periods.
