We had the opportunity to talk with HP’s KC Choi, Vice President, Global Solutions Architecture, Sales, Technical and Partner Enablement, about big data as it pertains to IT and being relevant to the business. Big data has been a buzzword everyone’s talking about but not everyone’s sure how to get into it. Regardless, it feels like we’re at a pivotal point where adoption is about to become mainstream.
Duncan: Big data is going to be extremely relevant to organizations. What is different about big data right now?
KC Choi: I think one of the fundamental differences is big data is actually becoming affordable and achievable using standardized technology and software. We’ve been talking about big data concepts, across different industries for many years, but I think the limiting factor has been whether you had enough horse power, development tools and applications to tap into the business benefit. We’re starting to move out of the esoteric realm into something that is not only affordable but easier to do and consume.
Duncan: What is HP’s thinking behind making big data affordable and accessible?
KC Choi: I think that big data or data warehousing used to be viewed as a tool for the chosen few, and teams spent a lot of time building custom solutions.
- Build it yourself – if you determine that you need to build it yourself, perhaps because of the industry you’re in and the compliance requirements related to that – look for technologies that are standard, re-deployable, well understood, and NOT closed, proprietary or single purpose.
- Consume it as a service – with the advent of analytics-as-a-service, we see a huge popularity for things such as Haven, which is a big data platform that we offer as a service. With this type of consumption model, companies aren’t limited by talent or budgets like they may have been 10 years ago. They can consume it in a hybrid model with some aspects in house and some as a service. I think we are going to continue to see this type of model accelerate.
Duncan: What should we be doing to be ready for the growing volume of data from a corporate IT perspective, in terms of making big data real?
KC Choi: I think that we need to take it in steps here because we are seeing unprecedented growth in the ability to generate data, and this is due to a number of forces:
- The mobility factor – consumers are elevating the number of devices and the number of config points into that data. The rise of the mobility phenomena is a huge factor in data growth.
- Unstructured data – the type of data that is generated today is very different to what we’ve seen in the past which has largely been highly structured, easy to be classify, and fits well in a single form. Big data sources are often unstructured, or have structures foreign to our current toolsets making them difficult to deal with in traditional means.
- The Internet of Things (IoT) – we are starting to see the next wave of information being generated from machine to machine traffic – we like to call it the Industrial Internet of Things. This data will be coming from completely new and foreign sources such as power systems, roadways, and other public infrastructure.
- Combined data – the combination of these data types – public consumer grade and industrial – to provide specific business outcomes or solutions is what is needed. A good example of this is to look at what’s happening with efforts around autonomous driving that’s going to probably impact our lives within the next few years. This is technology that has typically been in closed ecosystems, but is now combined with publicly available networks, roadways, and throughways to be useful.
- Making sense from it all – The value lies in understanding this marriage effect that needs to deal with not only the volume of data, but the variety of it too. How we correlate it and make it useable in ways that we haven’t even started to tap into is going to have a huge impact in the coming years.
Duncan: What advice do you have for people such as enterprise architects to adapt and stay relevant, who see this groundswell of change coming?
KC Choi: We grew up in an era where we looked at data in a very contained, structured way and queried it in a defined way with a language that we all spoke. That has fundamentally shifted.
- Re-educate – we’ve all got to get re-educated in a lot of respects because the way data is managed, queried, stored, manipulated – the chain of custody of the data – is a fundamentally different language now. It’s much more open source based, less and less of what I call merchant type environments with off the shelf components.
- Platforms – we’re seeing a dramatically changing landscape around the type of platforms that process this information, compute against it and store it. One of the beliefs we have at HP is we’re due for a fundamental re-think because of this rise in the volume, variety, velocity and veracity of the data that is coming in. We don’t think that some of the tried and true ways of computing, storing and moving that information are adequate enough with the growth that we have.
Duncan: At Long View, we’ve started talking about big answers instead of big data which is really about focusing on what we’re trying to achieve with big data and driving outcomes. What are the right questions to drive a big data initiative?
KC Choi: I think we have been limited by the languages that we have been taught, the information, and the parameters that have traditionally been set. A good example of this is the structured query language that, by its very natures, means there is a very formulated way of getting very formulaic data. The way to get to this is to ask the right questions and advise the business as to what those questions are.
- A good way to start is to think about asking “what is possible?” And IT is in a good spot to answer that, because a lot of times the formulation or the outcome is going to be driven by what can we get out of the environment.
- Don’t start with the magic question that you’ve been trying to answer forever. Go back and look at the questions that you know you can answer, but may have been difficult or time-consuming to do so. Or look at those questions that might be a 6 on a difficulty scale of 10. Then start putting them in context of what is possible and what can be tied back to solving real business problems.
- Data visualization – the other element you really have to look at is the analytics and visualization of that data. Data visualization has evolved to a point where part of the success is determined by how you can actually represent those relationships.
Duncan: How do you control scope in big data projects?
KC Choi: I’ve seen these projects take a life of their own and what you’re going to find is that they are somewhat unpredictable.
- Be open to adjusting to different outcomes and results when taking on big data projects. Once IT and business groups start to see what’s feasible, the requirements definition is going to start to expand and I think you’ve got to leave yourself open to that.
It’s a brave new world from a platform and technology perspective, as well as the types of questions one can ask and the way you can visualize data. The degree of open-mindedness of your team is the key to using big data to its full potential.