Data Analytics: Practical Guide to Leveraging the Power of Algorithms, Data Science, Data Mining, Statistics, Big Data, and Predictive Analysis to Improve Business, Work, and Life

Chapter 2: Big Data
Across the globe, data and technology are interwoven into society and the
things we do.  Like other production factors – such as human capital and
hard assets – there are many parts of the modern economic activity that
couldn’t happen without data.  Big data is, in short, the large amounts of
data that are gathered in order to be analyzed.  From this data, we can find
patterns that will better inform future decisions.
This data and what can be learned from it will become how companies
compete and grow in the near future.  Productivity will be greatly
improved, as well.  Significant value will be created in the economy of the
world because of increase in the quality of services and products while
reducing waste.  While this data has been around, it has only really excited
people that are already interested in data.  As times have changed, we are
getting more and more excited by the amount of data that we’re generating,
mining and storing.  This data is now one of the most important economic
factors for so many different people.
In the present, we can look back at trends in IT innovation and investment. 
We can also see the impact on productivity and competitiveness that have
resulted from those trends and how big data can make large changes in our
modern lives.
Like the previous IT-enabled innovations, big data has the same
requirements to move productivity further.  For example, if you see
innovations in current technology, then there will need to be a close
following after of complementary management innovations.  Big data
technology supplies and analytic capabilities are so advanced now that it
will have just as much of an impact on productivity as suppliers of other
technologies.  Businesses around the world will need to start taking big data
seriously because of the potential it has to create some real value.  There are
already retail companies that are putting big data to work because of the
potential it has to increase the operating margins.

---

Big Data – A New Advantage
Since it has come to light, big data is becoming an incredibly important way
that companies are outperforming each other.  Even new entrants into the
market are going to be able to leverage strategies that data has found in
order to compete, innovate, and attain real value.  This will be the way that
all the different companies, new and established, will compete on the same
level. 
There are already examples of this competition everywhere.  In the
healthcare industry, data pioneers are looking at the outcomes of some
pharmaceuticals that are widely prescribed.  From the analysis of the
results, they learned that there were risks and benefits that had not been
seen in the limited trials that companies had run with the pharmaceuticals.
There are other industries that are using the sensors in their products to gain
data that they can use.  This can be seen in children’s toys, large-scale
industrial goods, and so many others.  The data that they gather show how
the products are used in real life.  With this data, companies can make
improvements on the products based on how people are really using them. 
This will make these products so much better for the future users. 
Big data is going to help create new growth opportunities and create new
companies that specialize in aggregating and analyzing data.  There’s a
good proportion of companies that will sit right in the middle of flowing
information.  They’ll be receiving information and data that comes from
many sources just to analyze it.  Managers and company leaders that are
thinking ahead need to start creating and finding new ways to make their
companies capable of dealing with big data.  People that do so will need to
be especially aggressive about it.
It’s important to realize that not only the amount of big data but the high
frequency and real-time nature of data as well.  There’s the idea of
“nowcasting” around right now.  This process is estimating metrics right
away.  These metrics can be things like consumer confidence.  Knowing
that information so soon used to be impossible and only something that
could be done after a while.  “Nowcasting” is being used more and more,
adding a lot of potential to the ways that companies predict things.

---

The high frequency of the data will allow users to try to test theories and
analyze the results in ways that they were incapable of before.  There have
been studies of major industries that have found ways that big data can be
used:
1. Big data can unlock serious value for industries because it makes
information transparent.  There is a lot of data that isn’t being recorded and
stored.  There is still a lot of information that cannot be found as well. 
There are people that are spending a quarter of their time looking for
extremely specific data and then storing it, sometimes in a digital space. 
There’s a lot of inefficiency in this work right now.  More and more
companies are storing data from transactions online, these people are able
to collect tons of accurate and detailed information about everything.  They
can find out inventory and even the number of sick days that people are
taking.
Some companies are already using this data collection and analysis to do
experiments and see how they can make better-informed management
decisions.  Big data allows companies to put their customers into smaller
groups.  This will allow them to tailor the services and products that they
are offering.  More sophisticated analytics are also allowing for better
decision making to happen.  There are fewer risks and bring light to
information and insights that might not have seen the light of day.
Big data can be used to create a brand new generation of services and
products that wouldn’t have been otherwise possible.  Some manufacturers
are already using the data that has been collected from their sensors to
figure out more efficient and useful after-sales services.

---

Big Data Creates Value
Using the US healthcare system as an example, we can look at ways that
big data can really create good value.  If the healthcare system used big data
to use the efficient and quality of their services, they would actually create
$300 billion of value every year.  70% of that value would have been seen
from a cut in expenditures.  These expenditures that would be cut are only
8% of the current expenditures.
If you look at European developed economies instead, you can see a
different way that big data creates value.  The government administrations
could use big data in the right way to improve operational efficiency.  That
would result in about €100 billion worth of value every year.  This is just
one area.  If the governments used advanced analytics and boosted tax
revenue collection, they would create ever more value just from cutting
down on errors and fraud in the system.
Even though we’ve been looking at companies and governments so far, they
aren’t the only ones that are going to benefit from using big data.  A
consumer will benefit from this system as well.  Using location data in
specific services, people could find a consumer surplus of up to $600
billion.  This can be seen especially in systems and apps that use real-time
traffic information to make smart routing.  These systems are some of the
most used on the market and they use location data.  There are more and
more people using smartphones.  Those that have smartphones are taking
advantage of the free map apps that are available.  With an increase in
demand, it’s likely that the nmber of apps that use smart routing are going
to increase.
By the year 2020, more than 70% of mobile phones are going to have GPS
capabilities built into them.  In 2010, this number was only 20%.  Because
of the increase in GPS capable devices, we can expect that smart routing
will have the potential to create savings of around $500 billion in fuel and
time that people will spend on the road.  That amount of money is equal to
around 20 billion driving hours.  It’s like saving a driver 15 hours a year on
the road.  This would save them $150 billion dollars in fuel.

---

While we have seen specific pools of data in the examples listed above, but
big data has a huge potential in combined pools of data.  The US healthcare
system is a great way to look at the potential future of big data.  The
healthcare system has four distinct data pools: clinical, medical,
pharmaceutical products; research and development; activity and cost; and
patient data.  Each data pool is captured and managed by a different portion
of the healthcare system.
If big data was used to its full potential, then the annual productivity of the
healthcare system could be improved around 0.7%.  But it would take the
combination of data from all these different sources to create that improved
efficiency.  The unfortunate part is that some of the data would need to
come from places that do not share their data at scale right now.  Data like
clinical claims and patient records would need to somehow be integrated
into the system. 
The patient, in turn, would have better access to more of their healthcare
information and would be able to compare physicians, treatments, and
drugs.  This would allow patients to pick out their medications and
treatments based on the statistics that are available to them.  However, in
order to get these kinds of benefits, patients would have to accept a trade for
some of their privacy.
Data security and privacy are two of the biggest roadblocks in the way of
this.  We must find a way around them if we really ever want to see the true
benefits of using big data.  The most prevalent challenge right now is the
fact that there is a shortage of people that are skilled in analyzing big data
properly.  By 2018, the US will be facing a shortage of 140,000 and
190,000 people with training in deep analysis.  They’ll also be facing a
shortage of roughly 1.5 million people that have the quantitative skills and
managerial experience needed to interpret the analyses correctly.  These
people will be basing their decisions off of the data.
There are many technological issues in the way as well that will need to be
resolved before big data can be used effectively by more companies.  There
are so many incompatible formats and standards that are floating around as
well as legacy systems that are stopping people from integrating data and
from using sophisticated analytical tools to really look at the data sets.

---

Ultimately, there will have to be technology made for computing and
storage through to the application of visualization and analytical software. 
All this technology will have to be available in a stack so that it is more
effective.  In order to take true advantage of big data, there has to be better
access to data, and that means all of it.  There are going to be so many
organizations that will need to have access to data stores and maintained by
third parties to add that data in with their own.  These third parties could be
customers or business partners.
This need for data will mean that companies that really need data will have
to be able to come up with interesting proposals for suppliers, consumers,
and possibly even competitors in order to get their hands on that data.  As
long as big data is understood by governments and companies, the potential
it has to deliver better productivity will ensure that there will be some
incentive for companies to take the actions that they have to get over the
barriers that are standing in the way.  In getting around these barriers,
companies will find new ways to be competitive in their industries and
against individual companies.  There will be greater productivity and
efficiency all around which will result in better services, even when money
is tight.
Big Data Brings Value to Businesses Worldwide
Big data has been bringing value to business internationally for a while. 
The amount of value that it will continue to bring is almost immeasurable. 
There are several ways that the big data has impacted the world so far. It
has created a brand new career field in Data Science. Data interpretation has
been changed drastically because of big data.  The healthcare industry has
been improving quickly and considerably since they added predictive
analytics into part of their business.  Laser scanning technology is changing
and has changed the way that law enforcement officers reconstruct crime
scenes.  Predictive analytics are changing how caregivers and patients
interact.  There are even data models that are being built now to look at
business problems and help find solutions.  Predictive analytics has had an
impact on the way that the real estate industry conducts business.

---

Big Data is a Big Deal
Besides the fact that data is bringing so much value to so many different
companies and industries, it is also opening up a whole new path of
management principles that companies can use.  Early on in professional
management, corporate leaders discovered that one of the key factors for
competitive success was a minimum scale of efficiency.
Comparatively, one of the modern factors for competitive success is going
to be capturing higher quality data and using that data with more efficiency
at scale.  For the current company executives that might be doubting how
much big data is going to help them, there are these five questions that will
really help them figure out how big data is going to benefit them and their
organizations.
What can we expect to happen in a world that is “transparent” meaning that
data is readily available?
Over time information is becoming more accessible in all sectors.  The fact
that that data is coming out of the shadows means that organizations, which
have relied heavily on data as a competitive asset, are potentially going to
feel threatened.  This can be seen especially in the real-estate industry. 
The real-estate industry has typically provided a gateway to transaction data
and a knowledge of bids and buyer behaviors that haven’t been available
elsewhere.  Gaining access to all of that requires quite a bit of money and
even more effort.  In recent years, online specialists are bypassing the
agents to create a parallel resource for real-estate data.  This data is gotten
directly from buyers and sellers, and available to those same groups.
Pricing and cost data has also seen a spike in availability for several
industries.  There are even companies using satellite imagery that is
available at their fingertips.  They’re using processing and analysis to look
at the physical facilities of their competitors.  That information can provide
insights into what expansion plans or physical constraints that their
competitors are facing.  But with all that data there comes a challenge.  The
data is being kept within departments.  Engineering, R&D, service
operations, and manufacturing will have their different information and it
will be stored in different ways depending on the department.

---

However, the fact that all this information is kept in these little pockets
means that the data cannot be used and analyzed in a timely manner.  This
can cause all sorts of problems for companies.  For example, financial
institutions don’t share data across departments like money management,
financial markets, or lending.  This segmentation means that the customers
have been compartmentalized.  They don’t see the customer across all of
these different areas, but just as separate images.
Some companies in the manufacturing business are trying to stop this
separation of data.  They’re integrating data from their different systems
and asking their smaller units to collaborate in order to help their data flow. 
They’re even looking for data and information outside of their groups to see
if there’s anything else out there that might help them figure out better
products and services.
The automotive industry has suppliers all around the world making
components that are then used in the cars that they’re making.  Integrating
data across all of these would allow the companies and their supply chain