Category Archives: Big Data

Back to dangerous blogging

May 20, 2015 9:51 pm / Leave a Comment /

Looking at this I have not written a blog post on here in over two years! I think it is time for a change. First of all to let you guys know what I have been up too and then too let you in what I will blogging about in the future:

What I have been up too

The past two years have been very interesting the company that I work for has seen a data explosion and unfortunately this meant that the DB2 estate that is needed to support this is quite expensive in terms of licensing and hardware costs compared to other offerings that are out there. Also we were struggling to find people with the requisite skills that supported our architecture of DB2 with an in house built ETL tools. It was rather unique solution in an area of the country were London sucks a lot of the people with DB2 skills.

Therefore we started looking at other platforms to house our data warehouse and power the businesses data future. We started looking at SQL server and all its offerings. This seemed like a winner for a time we started and building out the platform with an home built auto runner and other interesting features that I may blog about later. This solution was far cheaper in terms of licensing than DB2 (sorry IBM) and also came with a lot of features such as SSRS and SSIS and also a more readily available talent pool in the local area. This architecture still had one major problem hardware, we fill it up and stress it out far quicker than our normal 5 year planning caters for.

Then Amazon brought out Redshift. This is a winner it has features that will support our businesses data growth and has an eco system of products round it that will support the businesses data hunger. Although it does not come with an ETL tool like SSIS; in an effort to standardise opposed to our current scripted batch system we found Talend. AS a reporting solution found an awesome product called Looker. We are also looking at other Big Data solutions such as Hadoop, EMR, Databricks and many other things data related.

What will I be blogging about the future

So I now intend to get back to my blogging about my solutions to technical issues we face and solve, cool features we use and you can you to around Amazon Redshift, Looker, Talend and maybe occasionally DB2 as we wind down our usage. I will also be blogging more about the conferences I have attended, big data solutions we try out. So watch out for more content and hopefully interesting articles and findings.



Posted in: Amazon, Big Data, Redshift / Tagged: , , , , , ,

Just finished reading: Customer Experience Analytics

January 29, 2012 9:36 pm / Leave a Comment /

This is another excellent free ebook from IBM that renders well on a kindle. This book can be down loaded from the IBM Information Management Book store  or the direct link here.

Overall like the Understanding Big Data book I reviewed last time I think it is a good introduction to the subject matter giving you a quick way to get up to speed with some of the concepts involved and the evolution going on around the social sphere and customer experience. The book is again split up into sections, this time three: Part One: The CEA Opportunity, Part Two: The Customer Experience Analytics [sic] Solution and Part Three: How to Package a Customer Experience Analytics [sic] Program.

Part One: The CEA Opportunity covers a few case studies of how various industries use customer experience to fuel decisions that affect the business and the customer.   It then moves on to how that our societies are moving toward increasingly automated way of interacting during the sales and marketing processes makes collecting the data for CEA a lot easier and quicker to act upon. The third chapter in this part looks at the evolution of the customer decision making process, and how a single customers influence on the wider world can (should) affect how a business deals with them. This raised some interesting thoughts in that basically people that are “listened” to (facebook, twitter, text messages in a social group) should be treated differently when they have a complaint than those that “listen” and do not contribute back, pushing “stardom” down onto those that are not famous, but are popular in a social group. The final chapter in this section looks at the “bazaar” of data that exists for CEA and touches on big data concepts again.

Part Two: The Customer Experience Analytics [sic] Solution is a slightly technical, but more theoretical look at with out pushing any particular products how you would go about creating you CEA solution. It covers Master Data Management (MDM), Stream computing, Predictive Modelling and a couple of other topics, but not to a depth to make you a master of these areas but at least enough to let you in on the conversation.

Part Three: How to Package a Customer Experience Analytics [sic] Program is basically how you would put together a business case for CEA and the conclusion of the book. The business case for CEA varies from needed to stay in business (mobile phone compaines) to currently only done on an Ad-Hoc basis and needs to be built up in the company or the industry.  It would be hard to place the company that I currently work for on this scale as I am un-aware of what and if anyone else does in the sector that we are in, but I think it has legs and should be something that we should be pushing, would defiantly like to get involved in the technical side.  I also think what we do have in place is to rigid in the way it carries out its current matching and we really need to be pulling in or getting the social sphere of the customer somehow.

Posted in: Big Data, BigData Case studies, Book read, IBM, Information Managment, InfoSphere Streams, Kindle, MapReduce, MDM / Tagged: , , , , ,

Just finished reading: Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data

January 7, 2012 9:33 pm / Leave a Comment /

I have just finished reading this book, I was excited about the IBM offering and the concepts around big data at IDUG, but after reading the book I want to find a project I can try this out on. The book can be downloaded from here: Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data.

The book is in two parts, Part 1: Big Data from the business prospective and Part 2: Big Data from the technology prospective. The first part of the book as it suggests does not touch on the technical aspects of big data only the benefits to businesses and how we all are already part of the Big Data world. The second part of the book explains at a high level all the different parts of the Hadoop cluster and how you get data in and out and process data in there. The second part also explains the IBM offering into this marketplace in the form of IBM InfoSphere BigInsights and Streams.

The as a high level description first part introduces the concept of the three V’s of big data, Volume, Velocity and Variety, the uses of these V’s in a number of different scenarios all of which are very interesting and I can easily see how it would bring you competitive advantage (probably the point of the case studies). The second part is for the techies explaining what Hadoop is and all of the different parts that make it up with MapReduce, common components and the file system. Also explaining all the other technologies surrounding Big Data such as Hive, Flume and Jaql.

So this is just a very light overview of the book, and well worth a read. I did it on my kindle, sometimes the text varies from page to page as it gets resized but overall it was fine.

Posted in: Big Data, BigData Case studies, Flume, Hadoop, IBM, InfoSphere BigInsights, InfoSphere Streams, Jaql, MapReduce / Tagged: , , , , ,

My Christmas Present is Ace

January 5, 2012 1:48 pm / 1 Comment /

So for Christmas my wonderful fiancée for Christmas bought me a new kindle Kindle, Wi-Fi 6″  and this has enabled me to do a lot more reading of all those ebooks and pdfs that you never realy get round to during the day and the laptop is too big for bed.

I am finding it great, I don’t actually have any “kindle” books on it only pdf’s but is is still great. With the pdf’s (presentations from IDUG this year and IBM redbooks and IBM educational books) depending on the quality of the all I have had to do so far is change the contrast, the kindle works out what size to display on screen although you can change this if you want. It remembers where you have go to even if you move off to another book and you can still bookmark. As they are not kindle books then it does not push the info around different kindle accounts but otherwise I cant fault it so far.

In future expect far more reviews of books and the ideas that they put across in future postings. I am going to start with a big data book that I picked up in hard copy format at IDUG, but have only just got round to reading it recently.

And thank you again honey for such a great gift.

Posted in: Big Data, Blogging, Book read, EMEA, IBM, IDUG, Kindle / Tagged: , , , , , ,

IDUG – EMEA – 14th – First real day

November 14, 2011 11:20 pm / Leave a Comment /

As a first time attendee and a goody two shoes I went to the orientation session after my breakfast which was nice again, although the bettered Cauliflower was a little strange though. The first session that I went to was:

Pro Active DBA – Michael Tiefenbacher

This was good and very informative and cant wait to take the bull by the horns when I get back and be pro active and set my own KPI’s that I can monitor and get hold off as opposed to something that I cant monitor easily set by my managers. I found this session easy to take notes in as the pace and content was good and was a lot like being back at university with the learning pace. I learnt from this session:

  1. That although I do not have licences for Workload Manager (WLM) I can still use some of the monitors for free after I have activated them and the database
  2. There is a LAG function in DB2 that takes two timestamps.
  3. There is an automated scheduler that “replaces” the task centre that has been depreciated, but it will only run one sotred procedure at a time

The second session I went too was:

Eliminating Performance Bottlenecks in DB2 9.7 and PureScale – Steve Rees

This session was presented at a pace quite a lot faster than the last, with code samples that were quite large so I am sure that they will be useful. It concentrated on “time spent” metrics, and the use of MON_* table functions to iterate through a bottle necks. From this I learnt:

  1. There are three levels of monitors: System, Data Object and Activity —> SQL statements.
  2. db2 +c disables auto commit
  3. Some of the MON_* stuff contains XML that can be interrogated to get even more information to a greater level of detail

There were then the “Vendor Solution Presentations” these were not well advertised to what they were actually about unless you were in the know or it was obvious from the title. I took the decision to not go to one of these and do an exam instead. Good plan in practice, but unfortunately when speaking to the IBM rep on the exam desk it seems the system was “broken”!! I did get a free book out of it though on Big Data and IBM offerings in this area so should be good for a read.

The final session that I went to  was:

Sneak peek at the future of DB2 application development – Leon Katsnelson 

This was an amazing and eye opening session and I defiantly do not want to be part of a company that sticks it head in the sand about the changes that are going on in the market place, with Social, Mobile, Cloud , consumerization of IT and big data becoming very important. Also the session came back on topic with the announcement of DB2 Express – C 9.7.5 being made available today. From this session I learnt:

  1. IBM has a big data offering with a basic (express – C) version and an enterprise version
  2. InfoSphere Big Insights will connect to DB2 -woot. Heres the good bit you can query the unstructured data from DB2 SQL!!!!
  3. Russia has 151% mobile penetration, meaning every other person in Russia has more then one mobile!!
The evening was great with Triton in the drinks reception (with nibbles) in the posh restaurant in the hotel. It was good, got to meet some other DB2’ers and discuss the challenges that we all face. All the iPads have gone now, someone managed to do the Layar quiz that was set and I did not win the lucky did so they have all gone . I was also force to finish 5 puddings at the end as there were many left! Thank you Triton and Julian, James and Iqbal for the free beer  and food.
So tomorrow seems like it will be another busy day starting at 0830 with the first talk! So better get to bed soon, tomorrow I want to go to:

Tuesday, November 15, 2011

08:30 AM – 09:30 AM
Session 4
09:45 AM – 10:45 AM
Tuesday VSP
11:00 AM – 12:00 PM
Session 5
12:00 PM – 01:00 PM
Tuesday Lunch
01:00 PM – 02:00 PM
Session 6
02:15 PM – 03:15 PM
Session 7
03:15 PM – 03:45 PM
Tuesday Coffee Break
03:45 PM – 04:45 PM
Session 8

Although I may change the last session to D8: Stuffed with great enhancements.

So may see some of you bright and early for breakfast, good night.
Posted in: Big Data, DB2, DB2 Administration, DB2 Maintenance, EMEA, IBM, IDUG, InfoSphere BigInsights, Triton