Exercise: Data Preparation

The following data issue is related to which data type?

1. A healthcare database contains patient records with incorrect medication dosages due to data entry errors. This can lead to incorrect treatment plans and potentially harm patients.

Answer: AccuracyTimelinessConformalityCompletenessValidityConsistencyUniqueness

2. A customer relationship management (CRM) system has numerous records without email addresses. This makes it difficult to reach out to customers for marketing campaigns or support follow-ups.

Answer: UniquenessCompletenessConsistencyValidityAccuracyTimelinessConformality

3. In a financial database, some transactions are recorded with different formats for the same currency (e.g., $100, 100 USD, 100.00$). This inconsistency complicates financial analysis and reporting.

Answer: CompletenessUniquenessConformalityConsistencyAccuracyTimelinessValidity

4. A logistics company updates its delivery status information with a delay of 24 hours. Customers checking the status of their shipments receive outdated information, leading to frustration and increased support calls.

Answer: ValidityAccuracyUniquenessCompletenessConformalityConsistencyTimeliness

5. An e-commerce platform has a product catalog where some items have invalid category codes that do not match the predefined categories in the system. This causes issues in inventory management and product search functionality.

Answer: CompletenessUniquenessAccuracyTimelinessConsistencyValidityConformality

6. A university student database contains duplicate records for some students due to variations in the way their names were entered (e.g., John Doe, John A. Doe). This duplication causes problems in tracking academic performance and financial aid distribution.

Answer: ConsistencyUniquenessCompletenessValidityTimelinessAccuracyConformality

7. In a company’s HR system, employee IDs are supposed to follow a specific format (e.g., EMP12345), but some records do not conform to this format (e.g., 12345EMP, EMP-12345). This non-conformity complicates data integration and reporting processes.

Answer: TimelinessConformalityCompletenessUniquenessAccuracyConsistencyValidity

Match the data error with the method to fix it.

1. If the proportion of absent entries is small, consider removing those records.

Answer: Missing ValuesInconsistent FormattingInconsistent Data TypeDupplicate

2. Fill in absent entries with a substitute value, such as the mean, median, mode, or a specific value. Advanced methods include using machine learning models to estimate the absent values.

Answer: Missing ValuesInconsistent FormattingInconsistent Data TypeDupplicate

3. Retrieve the absent entries from external sources or databases where possible.

Answer: Missing ValuesInconsistent FormattingInconsistent Data TypeDupplicate

4. Standardize the format and convert entries to this standard.

Answer: Missing ValuesInconsistent FormattingInconsistent Data TypeDupplicate

5. Ensure entries are in the correct type and convert them as needed.

Answer: Missing ValuesInconsistent FormattingInconsistent Data TypeDupplicate

6. Remove repeated records.

Answer: Missing ValuesInconsistent FormattingInconsistent Data TypeDupplicate

G. Merge repeated records into a single entry.

Answer: Missing ValuesInconsistent FormattingInconsistent Data TypeDupplicate

Data normalization, data standardization, data encoding?

What is the meaning of this formula?

\[ x_{n e w}=\frac{x-x_{\min }}{x_{\max }-x_{\min }} \]

Answer: Process of transforming data into a scale between 0 and 1 using max and min using.Process of transforming data into a scale between 0 and 1 using mean and standard deviation (std).Process of transforming data into a scale between -1 and 1 using max and min.Process of transforming data into a scale between -1 and 1 using mean and standard deviation (std).Process of transforming data using mean and standard deviation (std)

What is the meaning of this formula?

\[ x_{\text {new }}=\frac{x-x_{\text {mean }}}{s t d} \]

Answer: Process of transforming data into a scale between 0 and 1 using max and min.Process of transforming data into a scale between 0 and 1 using mean and standard deviation (std).Process of transforming data into a scale between -1 and 1 using max and min.Process of transforming data into a scale between -1 and 1 using mean and standard deviation (std).Process of transforming data using mean and standard deviation (std)

3. Which one is the best explanation of data encoding?

Answer: Process of combining data from two sources.Process of converting data that is in the wrong type.Process of removing data that is in the wrong type.Process of converting data from one data type to another.Process of creating backup copies of data for safekeeping.Process of analyzing of data to identify patterns and trends.Process of converting text into images to make it more visually appealing.Process of combining multiple datasets into a single, consolidated file.

4. Which of the following are reasons for performing data transformation techniques such as encoding, normalization, and standardization? Select all that apply.

Choices: TRUE or FALSE

4.1 Ensures all features contribute equally to the model. TRUEFALSE

4.2 Enables the use of categorical data in numerical algorithms. TRUEFALSE

4.3 Accelerates the convergence of gradient-based algorithms. TRUEFALSE

4.4 Reduces the size of the dataset for faster processing. TRUEFALSE

4.5 Improves the accuracy and performance of machine learning models. TRUEFALSE

4.6 Makes data compatible with algorithms that require numerical input. TRUEFALSE

4.7 Ensures data follows a uniform distribution. TRUEFALSE

4.8 Facilitates easier comparison and interpretation of features. TRUEFALSE

4.9 Eliminates the need for further data preprocessing. TRUEFALSE

4.10 Reduces the need for feature engineering. TRUEFALSE

Give the data from this link

Fill in the following table.

Store Location	Female	Male	Grand Total
Chiang Mai	Q1)	Q2)	4240031
Lampang	2113791	Q3)	Q4)
Lampoon	Q5)	2136273	Q6)
Grand Total	Q7)	Q8)	12833270

Based on the given data, answer the following questions:

1. How many entries are for males?

2. How many entries are for males at the Chiang Mai location?

3. What is the average salary of females?

4. What is the average salary of females in Lampang?

5. What is the average salary of customers aged between 25 and 35 (inclusive)?

6. What is the average salary of customers aged over 25 and up to 35?

7. What is the average salary of customers aged between 25 (inclusive) and less than 35?

8. What is the average salary of customers aged over 25 and under 35?

X	X Normalized	X Standardized
66	0.59	0.48
36	0.29	-0.48
87	0.80	1.15
7	0.00	Q1
33	0.26	-0.58
40	0.33	-0.35
100	Q2	1.56
48	0.41	-0.10
62	0.55	0.35
35	0.28	-0.51
20	0.20	-0.99
18	0.13	-1.05
85	Q3	1.08
95	0.85	1.30
92	0.81	Q4
88	0.82	1.30
32	0.25	-0.61
98	0.92	1.53
16	0.09	-1.12
9	0.02	-1.34

1. From the variable X, what is the minimum value?

2. From the variable X, what is the maximum value?

3. From the variable X, what is the average value? (Use 2 decimals.)

4. From the variable X, what is the SD value? (Use 2 decimals.)

From the the above table.

Q1 = (Use 2 decimals.)

Q2 = (Use 2 decimals.)

Q3 = (Use 2 decimals.)

Q4 = (Use 2 decimals.)