Replacing Data Engineering with Automation | Watch the Recording
What are your views and experiences of replacing data engineering with automation?
In this recording, Matt talks through what Business Intelligence (BI) comprises and utilising it as an asset, including:
- reporting and dashboarding
- analysis and insight
- modelling and forecasting
- Machine Learning, AI, data science and data products
He emphasises the value or organising your data, exposing any gaps and its ability to empower analysis.
Matt addresses the data challenge, including siloes, pipes, schedules and warehouses. He points out that data is orders of magnitude which are more powerful when combined together. He states that building a single source of truth, centralising the logic and uniting everything is critical.
Matt talks through what the solution looks like through demonstrating a diagram which shows how to ‘extract and load’ and ‘transform’ by applying data modelling. He explains the different options for getting the data piped in and centralised, through using data engineers, a consultancy or looking to modern automation with analysts and BI tooling.
Matt is an advocate of automating the repeatable and moving people to the place in the business where they add the most value, emphasising that people are better than software – freeing data engineering up and moving it to where it’s really needed. He talks through his experiences of working at thetrainline.com, Just Eat and momondo to advise on data engineering, where it is needed and how this inspired him to build kleene.ai
“I think that by combining your software engineering with your data engineering in those spaces, that’s where you well and truly become a data company. It’s where data is the product itself or it’s certainly leveraging the product to its full capability.”
Do you love the idea of empowering analysts to own the space or are you more about leveraging individual engineering technologies and teams to continue to own the space and build something bespoke?