As mentioned in the previous post starting this all up again then I have been looking at expanding and improving my skills around python and Google Cloud.
What did I do:
To this end I took up Googles offer and signed up for the “Free courses during COVID” offer and followed through with the QwickLabs modules. One of the quests contained a IOT device simulator, this was a fairly simple amount of data that was passed through all the way to Google Big Query. I am going to take this and completely plagiarise it as a learning example to build upon as I think it is a good basis for a lot of things:
- Improving my python – The first iteration I will publish I plan to replace the data files the code ingests with something that generates “random” data. There is a lot of scope to use different methods to improve this
- Extending the pipeline – This pipeline can go all the way through to visualisation in Google Data Studio
- Looking at Google Big Query – This is a very interesting area, we can look at functions, GIS and all things GBQ
- Other Google Services – There are many services used in this example and I feel that we can add more as we need such as Google Cloud Composer,
Where can this wonder code be found:
I have a Git Hub account where I have various white whales that I have started, and this particular one can be found: here
How is this going to work:
I am going to start by creating a pipeline that is not much of a departure from what is offered by the current Quest. I will then iterate on that to produce proof of concepts and give appraisals of what I have done, try and critique myself. You will be able to find the work in my GitHub repo and we can see where we go from here, depending on mainly when I get time to do these things!
Who can help:
You all can if you think that there is a better way to do literally everything I would love to know and investigate. I am pretty certain that there is for my Python location data generating stub after the first rushed iteration.
I look forward to hearing from you all!