It’s been a while ….. again:
So since I last posted way back in 2016 and QCon there has been a lot of water under the bridge in terms of technologies used and abused, and either taken into the stack or discarded to the ummm it nearly worked pile.
Things that have changed in that time:
Where a I work has seen a massive increase in the number of people in the data team in response to the business becoming far more data driven. The team has gone from two or three of us to three distinct sections all with a minimum of four people in them.
We now embrace what was just emerging as a widespread phenomenon in 2016 – streaming data; and make good use of this for micro services and data products.
I now have processed to a “senior” role in the business; with juniors and none prefixed or suffixed data engineers in the data products team.
Things that have NOT changed in that time:
We are still moving off DB2 but due to several large scale events over the years and the increasing size of the team then this is rapidly gathering pace but as a business we still need the old beast, but it has been on somewhat of a diet!
Where are things going:
Well for us from the Data Centre and Amazon Web Services (AWS) and to Google Cloud. Making extensive use of everything that it has to offer because of it nature we face far less of the size issues that we did on the data centre or even on AWS with some of its offerings
What are my interests (white whales) now:
Well I am going to be looking at a few things now:
- Google Cloud Management – is it possible to create a tool that will aid this from a green field prospective and also brownfield. It seems (not that I can google so they need to do a better effort at SEO) that there is not a product out there that does this in an obvious way
- IOT – This is going to be something that is “easily implemented” but then has individual elements that can taken WAY further, currently working on something and look put for upcoming posts