Data Work

  Replication is the process of copying data from a database or a centralized warehouse to another storage repository. The source from where the data is moved is called the publisher and the point to which it is moved by the replicating software is known as the subscriber. Since the data is only replicated at multiple locations, all users have access to the same day regardless of whether it is situated in the publisher or the subscriber. Typically, a file server called the disconnected subscriber links a subscriber to a remote location or a laptop. Now, why should you use replicating software? In the modern data-driven business environment, there is a need to have copies of their database as protection from data breaches or data loss. Even though the simple procedure is to have a snapshot of the database and store it, this solution does not take into account changes in the data occurring continually in the source database and hence, the backup is not updated. In such circumstanc...

AmazonDMS ( Database Migration Service) is an optimized cloud-based service that facilitates the migration of data between various types of source and target databases. These include on-premises servers to the cloud, one cloud provider to another, and between relational databases, data warehouses, NoSQL databases, and more. Using this service, organizations can either choose a single snapshot data migration or continuous movement of data between the source and the target databases provided both are continually kept in sync.  For data migration to function optimally with Amazon DMS, there are two pre-conditions that have to be fulfilled. The first is that a link has to be made between the source and the target database so that the software knows from where the data has to be moved and to which data storage repository. Once this step is done, select the activity that will load the data from the source to the target. After these steps are defined, Amazon DMS automatically initiates...

In the modern data-driven and digital business environment where organizations depend largely on data to power operating efficiencies and analytics, insulating classified data from breaches or during an outage has now become the prime concern. One way to do so is to have data backups but then, all changes and incremental data that occur continuously in the source database will not be taken into account and sent to the target database. The most optimized solution to get around these issues is to use advanced replicating software that continually updates data in the source database to the target database provided both are always kept in sync. For more information on replicating software, click here.  The main benefit of replicating software is its ability to replicate data to multiple sites so that users can access this data from any remote location. This is a very critical advantage during an outage in the primary server as users can access data in secondary servers that are...

Executing Change Data Capture (CDC) to a data lake on Amazon S3 from a relational database necessitates that data should be handled at a record level. All the files have to be read, the necessary changes made, and complete datasets have to be rewritten as new files by the processing engine to enable operations such as inserting, updating, and deleting specific records from a dataset. On the other hand, when AWS CDC to S3 provides the data in the data lake data in real-time, it is often fragmented over several small files. The resultant poor query performance can be partially resolved with Apache Hudi which is an open-source data management framework managing data at the record level in Amazon S3. The result is that with AWS CDC to S3, building CDC pipelines becomes a simple process that is optimized for streaming data ingestion.  You can also build a Change Data Capture pipeline with AWS DMS to capture data from an Amazon RDS for MySQL database. These changes may be applied to a...