Data Analysis From Scratch With Python: Step By Step Guide

Online Data Sources
We’ve discussed how to process data and select the most relevant features. But
where do we get data in the first place? How do we ensure their credibility? And
for beginners, where to get data so they can practice analyzing data?
You 
can 
start 
with 
the 
UCI 
Machine 
Learning 
Repository
(
https://archive.ics.uci.edu/ml/datasets.html
) wherein you can access datasets
about business, engineering, life sciences, social sciences, and physical sciences.
You can find data about El Nino, social media, handwritten characters,
sensorless drive diagnosis, bank marketing, and more. It’s more than enough to
fill your time for months and years if you get serious on large-scale data
analysis.
You 
can 
also 
find 
more 
interesting 
datasets 
in 
Kaggle

---

(
https://www.kaggle.com/datasets
) such as data about Titanic Survival, grocery
shopping, medical diagnosis, historical air quality, Amazon reviews, crime
statistics, and housing prices.
Just start with those two and you’ll be fine. It’s good to browse through the
datasets as early as today so that you’ll get ideas and inspiration on what to do
with data. Take note that data analysis is about exploring and solving problems,
which is why it’s always good to explore out there so you can be closer to the
situations and challenges.
Internal Data Source
If you’re planning to work in a company, university, or research institution,
there’s a good chance you’ll work with internal data. For example, if you’re
working in a big ecommerce company, expect that you’ll work on the data your
company gathers and generates.
Big companies often generate megabytes of data every second. These are being
stored and/or processed into a database. Your job then is to make sense of those
endless streams of data and use the derived insights for better efficiency or
profitability.
First, the data being gathered should be relevant to the operations of the
business. Perhaps the time of purchase, the category where the product falls
under, and if it’s offered in discount are all relevant. These information should
then be stored in the database (with backups) so your team can analyze it later.
The data can be stored in different formats and file types such as CSV, SQLite,
JSON, and BigQuery. The file type your company chose might had depended on
convenience and existing infrastructure. It’s important to know how to work with
these file types (often they’re mentioned in job descriptions) so you can make
meaningful analysis.

---