Should a data Engineer know in depth the models used by data Scientist? In principle, you should know what it means to use one or another model for the environment, and what architecture is ideal for them to work.
In summary, the data Engineer is in charge of the infrastructure of BIG data. How important can this be? According to Todd Goldman’s article, which is part of a Gartner study, proclaiming that only 15% of Big Data’s projects go to production, it is evident that basic implementations in architecture are overlooked.
This is the key to realizing why it doesn’t get to production that 85% is remaining. The slowness with which the data is loaded, the failure to do so automatically and incrementally, the inability to consult them and the little agility to migrate from the testing environment to the production are problems that the inclusion of more data engineers would help to solve.
The data Engineer plays a crucial role in converting a proof of concept of Big data into a real and palpable project. That is to say, from prototype to production.
At this point, many may wonder what a Data Architect would be. According to our point of view, data Architect is a data Engineer with a more global vision, and more oriented to the integration, centralization and maintenance of all the data sources.
We are aware that we may have left some profile that someone considers essential. The MIS reporting executive, the Business Analyst, the statistician, the Machine Learning Engineer, or even the Data Translator.
There are also traditional profiles such as Oracle DBA, Teradata Business Analyst that has been recycled and even have their function here. But with this article, we tried to talk more about the roles that are played in the world of Big Data and not profiles or titles.