Monday, June 18, 2018

What is Data Engineering

ata engineering ensuring all right data (internal/external, structured/unstructured) are identified, sourced, cleaned, analyzed, modelled, and decisions implemented — without losing on granularity and value as the data travels this path.
Data Engineering has to help businesses by building robust capabilities to deal with the volume, velocity, reliability, and variety of data and makes this data available for business users to consume — both as traditional marts and warehouses, and new-age big data ecosystems.
Data engineering is dealing with data — data lakes, clouds, pipelines, and platforms. Data Warehouse is the base of BI (Business Intelligence) project, and ETL (Extract, Transform and Load) is the base of Data Warehouse.

Data Approaches: There are many data engineering approaches which are very helpful to understand different techniques as given below-
1. Implement Data Lakes/ Data Warehouses/ Data Marts: Help lay or enlarge the enterprise data foundation so a range of analytics solutions can be built on top
2. Develop Data Pipelines: Facilitate production grade end-to-end pipeline of data-to-value that takes data solutions from sandbox environments, and rolls them out to end users
3. Enable Value Realisation: Data engineering enables data science solutions to reduce operational costs, discover new revenue sources, and create new products

Friday, June 1, 2018

SSIS - How to call multiple child packages by parent or master package

In our day to day activities, we can build a data extraction module that can be called from different packages. 
For example, we have to load the data into a star schema, and we can build a separate package to populate each dimension and the fact table and these packages are located in some folder. They should be executed in certain order one by one.
Now, we want to create one master SSIS package that will go to that folder and grab those child packages and execute or run them one by one (no repeat) in a predefined order.
To accomplished this, You could build a Foreach Loop Container with a Package Execute Task in it to execute them, then we would need to name them so that they were retrieved in the order we wanted. 
With the help of variables, we can set the package name property in Package Execute Task and this variable will get the next value from the Foreach loop Container.

Foreach Loop Container can get these package names from a data table that contained the order we wanted and then we populate an SSIS object type variable with the record-set and use that to feed the order and the list to a Foreach loop just like we mentioned above. The only difference would be the source of the list.

The below video is capable to explain that How can we call multiple child packages by parent or master package?

Popular Posts