Data Science Thoughts

  1. What is Hypothesis Testing and when do we use it? Hypothesis testing is a part of statistical analysis, where we test the assumptions made regarding a population parameter. It is generally used when we were to compare: a single group with an external standard two or more groups with each other A  Parameter  is a number that describes the data from the  population  whereas, a  Statistic  is a number that describes the data from a  sample . 2. Terminology used Null Hypothesis:  Null hypothesis is a statistical theory that suggests there is no statistical significance exists between the populations. Alternative Hypothesis:  An Alternative hypothesis suggests there is a significant difference between the population parameters. It could be greater or smaller. Basically, it is the contrast of the Null Hypothesis. Note: H 0  must always contain equality(=). H a  always contains difference( ≠,  >, <). For example, if we...

 -> We are going to have a deep discussion on the Z score. But right before that, we need to understand what a normal distribution and a standard normal distribution are. What is a distribution? A distribution in statistics is a function that shows the possible values for a variable and how often they occur. it may occur with various different values like age, height, the weight of people. What is a Normal distribution? The normal distribution is a distribution that is symmetric about the mean(mean is nothing but average of all the observations). Most of the observations in the normal distribution are surrounded by the mean. What is a standard normal distribution? The standard normal distribution is a normal distribution whose mean and standard deviation are scaled at 0 and 1 respectively. Z score can only be calculated for the observations which follow a normal distribution. What is a Z score? A Z-score is a numerical measurement that describes a value’s relationship to the mea...

Case study:   We Have a used cars dataset from the website. This dataset contains information about used cars. This data can be used for a lot of purposes such as price prediction to exemplify the use of linear regression in Machine Learning. The columns in the given dataset are as follows: name, year, selling_price, km_driven, fuel, distance, seller_type, transmission, Owner For used motorcycle datasets please go to  https://www.kaggle.com/nehalbirla/motorcycle-dataset Here using the above features we should predict the selling price of cars. so feature km_driven and distance are in different scaling if we load these features into a model then prediction may go wrong due to the wrong interpretation of slops. To overcome these we will scale down these features into normal values between 0 to 1. from  sklearn.preprocessing  import  MinMaxScaler Minscaler = MinMaxScaler() scaler = Minscaler.fit( 'distance', 'km_driven' ) scaler.data_min_ scaler...

In many of the supervised machine learning problem statement basically we have two kind of use cases. 1.Regression 2.classification For Regression type use case if we want to check accuracy usually we follow the techniques like R square and adjusted R square. In this article we will be discuss about difference between R square and adjusted R square. 1. R square R square formula is given by, = coefficient of determination = sum of square of residuals or error = total sum of square Here residuals are sum of square of difference between actual point and predict point. and for given data if we have only target variable we will find the best fit line by taking average of all the values. so total sum of square of actual points and average value. So from above formula we will get value between 0 and 1, the more value near to 1 is the best fit line.  whether can we get R2 value less than 0? YES, only when your best fit line is worse than average best fit line. If RSS > TSS so ratio beco...

  What is Variable? A variable is something that is need to be measured. it is a type of recorded piece of information or characteristic about a person or case or unit in our study. for Example : We might record the age of everyone in our sample and for every one we're recording the age but the age is going to change or vary from person to person so this is kind of the opposite of a constant which is always the same. for the sake of our discussion we have few examples like age , weight , BMI, does someone have disease, yes or no etc.. we summarize and analyze the data depend on type of variable we have.  Type of variables: 1. Independent variable : It is also known as experimental or predictor variable. Independent variable is a variable that is the causes or reason of any situation which can be manipulated. 2. Dependent variable : Dependent variable is something that depends on their factors. It is also known as outcome variable. for Example : Time spent studying causes a c...