Normalization and Standardization

Suppose if you have any use case, so the most important thing for the use case is data. Initially, you will be collecting the data so if you have collected the data that data have many features so those features may contain independent feature and dependent feature so with the help of the independent we will try to predict dependent feature in supervised machine learning.

so when you consider these features this has 2 important properties.

1. Unit 

2. Magnitude

let's have features like personage, height, weight, etc. so if I consider the feature age the unit basically no of years and the magnitude is basically value. 

For Ex: Suppose if I say 25years then 25 is magnitude and years is unit.

Each feature is calculated with unit and magnitude so if you have many features so it will get computed with different units. so this unit and magnitude vary between different features. so it is very necessary that for the machine learning algorithm the data we provide that we should try to scale down the data with a particular scaling value.

For this type of problem, we use 2 main techniques.

1. Normalization:  Normalization helps you to scale down your feature between 0 to 1.

2. Standardization: Standardization helps you to scale down your feature based on a standard normal distribution. Usually(Mean is 0 and Standard deviation is 1).

lets us discuss Normalization and Standardization.


Normalization(Min-Max normalization)

In this approach we will scale down the values of features between 0 to 1.

X norm = (X - X min) /(X max - X min) 

from sklearn.preprocessing import MinMaxScaler 

scaling = MinMaxscaler()

scaling.fit_transform(['age', Height'])

From above I am trying to scale down the age and height feature between 0 to 1. 
Sklearn has a library called MinMaxscaler which will transform the data by applying the above X norm formula to each and every element in age and height.


Standardization(Z-score normalization)

here all the features will be transformed in such a way that they will have the properties of a standard normal distribution on mean=0 and standard deviation=1.

Z = (X - Mean) / Standard Deviation

from sklearn.preprocessing import standardscaler
scaling = standardscaler()
scaling.fit_transform([age, height])

SK learn has a library called standard scaler which will transform the data considering which is mean =0 and standard deviation=1

This is the most popular used technique in most of the problem statements.


Comments

Post a Comment

Popular posts from this blog

Transformers: Self-attention

Image
 In the realm of neural networks, three primary types are commonly discussed: Artificial Neural Networks (ANNs) : These are fully connected networks comprising input, hidden, and output layers. Each neuron in a layer is connected to every neuron in the subsequent layer, enabling complex pattern recognition through weighted connections. Convolutional Neural Networks (CNNs) : CNNs incorporate convolutional layers with kernels (filters) that slide across the input data to detect features. These networks also perform pooling operations to reduce dimensionality and flatten the data before passing it to fully connected layers. CNNs are particularly effective for image and spatial data processing. Recurrent Neural Networks (RNNs) : RNNs are designed to handle sequential data by maintaining a memory of previous inputs through their hidden states. Unlike feed-forward networks, RNNs can process input sequences of variable length, making them suitable for tasks involving time series data, nat...
Read more

Retrieval Augmented Generation(RAG)

Image
 We Know core ideas like NLP, LLM. Let's look at how we might build utilizing LLM and the deployment aspect using RAG. Retrieval-Augmented Generation (RAG) is a technique in natural language processing that combines retrieval-based and generation-based approaches to create more accurate and informative responses. Let's break down each point: 1. RAG Architecture  Architecture: Retriever: Retrieves relevant documents or passages from a large corpus based on the input query. Reader/Generator: Uses the retrieved documents to generate a coherent and relevant response. Combiner: Integrates the retrieved information with the generated response. 2. Pipeline Behind the RAG The RAG pipeline is a collection of the above components working together: Query Processing: The input query is encoded into a vector. Retrieval: The encoded query is used to retrieve relevant documents from a vector database. Document Encoding: The retrieved documents are encoded into vectors. Information Inte...
Read more

Large Language Models(LLMs)

Image
  Language Model A language model is a type of artificial intelligence model designed to understand, generate, and manipulate human language. Language models are trained on large datasets of text to learn patterns in language use, including grammar, vocabulary, and context. Key functions include: Text Generation : Producing coherent and contextually relevant text. Text Completion : Predicting the next word or phrase in a given context. Machine Translation : Translating text from one language to another. Sentiment Analysis : Determining the sentiment expressed in a piece of text. Text Summarization : Creating concise summaries of longer texts. Language models can be based on various architectures, such as Recurrent Neural Networks (RNNs) and Transformer models. Large Language Model (LLM) A large language model (LLM) is a type of language model that is characterized by its large size, typically measured by the number of parameters (weights) it has. LLMs are trained on vast amounts ...
Read more