Data engineering is fighting over standards and governance, and it is not easy to align a large organization to a set of governing standards. You must choose the technology stack and tools that are appropriate for you, the company, and your requirements. If you are searching for ETL solutions for the enterprise, the following are some extra considerations-
- Market availability of skill set
- No code or low code
- Monitoring your pipelines has never been easier
- Model of licensing. It depends on the number of automobiles, memory, and so on.
Note: In your tooling, make a split in data
ingestion, data transformation and data storage and look for those 3 parts
separately.
For
example, you can work on a full open-source data platform with-
1.
Airbyte or Airflow for
data ingestion,
2.
dbt or DataForm for transformation and
3.
you can use a combination of Postgress, minio and clickhouse
for storage.
To
learn more, please follow us -
http://www.sql-datatools.com
To
Learn more, please visit our YouTube channel at —
http://www.youtube.com/c/Sql-datatools
To
Learn more, please visit our Instagram account at -
https://www.instagram.com/asp.mukesh/
To
Learn more, please visit our twitter account at -
https://twitter.com/macxima
No comments:
Post a Comment