KandH BD-101 Big Data Experiment Lab

What is big data?

Big data refers to a collection of large data sets with diversified structures.
In the past, big data was usually used for data analysis or statistical applications in enterprises, and it was limited to analyzing historical data.
Today, data analysis advanced from analyzing historical data to prediction of future trends with improved accuracy.
This is due to the development of clouds and the Internet of Things, lowered costs for storage equipment, rapid growth in data volumes, and the improvement in software technologies.
Big data is no longer just data processing, but an unprecedented tool for business intelligence.

1. System independence:
It can be operated without any internet connection nor any additional hardware/software installation. The cabinet design makes it easy to move.

2. Convenience:
The troubleshooting function provides system restoring function, so users can troubleshoot quickly. Through 6 different models of random data generation, users can generate datasets suitable for different algorithms easily.

3. Expansion:
It can be applied to different researches and experiments for big data analysis, and it can also be combined with IoT-enabled devices to store and analyze various data sets from sensors to achieve cross-domain applications.

4. Richness:
It provides comprehensive courseware for big data analysis
(1) 9 different algorithms & 20+ classic big data analysis examples
(2) Tools, such as Hadoop, Yarn, Spark, Hive, and HBase, are introduced and applied.

► Volume
In various fields such as financial services, energy management, biomedicine, and multimedia communities, a large number of data sets is being generatedevery second.

► Velocity
Whenever data is sent to servers, it will be analyzed immediately and bring real-time modifications on the previous results so as to obtain latest findings with maximum data values.

► Variety
Diverse data includes structured and unstructured data, such as text, location, sound, videos, and pictures, all of which can be interactively analyzed to reveal correlations between data sets.

► Veracity
Is the data source correct? Is the data accurately recorded even if it's true?
Are there any anomalies in data sets? Wrong data sources may result in deviations in the analysis results and affect the accuracy of the predictions.
Therefore, ensuring authenticity of data sources is also one of the key points of big data analysis.

► Value
The greatest values of big data analysis lie in digging out the data that is valuable for future trends from massive data, and performing in-depth analysis through artificial intelligence and machine learning to improve accuracy.

► Hadoop - The System Foundation of Big Data
Hadoop is an open source software framework that can successfully solve various problems, such as file storage, file backup and data processing. Therefore, it is widely used and has become the mainstream technology of big data analysis.

► Spark - Data Processing for Big Data Speed is very important in data processing when it comes to big data. An important feature of Spark is that it can be operated in memory, which makes Spark more efficient in data analysis and calculation than Map Reduce.

► Python - The Extraction of Big Data
Python is a programming language commonly used in various fields. It can crawl large amounts of effective data from the network in a low-cost and automated manner. The powerful data processing capability is the main reason why Python becomes an important programming language when analyzing big data.

► Data cleaning, regularization and standardization
► Architecture and configuration of the big data ecosystem
► Comparison of assorted databases
► Use various algorithms to extract, store, retrieve and analyze big data
► Integration of big data analysis and artificial intelligence