Least Squares

This is note1 based on Linear Algebra And Learning From Data by Gilbert Strang.

Least Squares

Many applications lead to unsolvable linear equations $Ax=b$. The least squares method chooses $\hat x$ to make $||b-A\hat x||^2$ as small as possible. Which is $(Ax-b)^T(Ax-b)$. Minimizing erorr means its derivatives are zero which leads to normal eqautions $A^TA\hat x=A^Tb$.

Four ways to solve

1.The SVD of A leads to its pseudoinverse $A^+$. Then $\hat x = A^+b$ :one short formula (This note)
2.$A^TA\hat x=A^Tb$ can be solved directly when A has independent columns
3.The... read more

Recently I was reading an article about a cool project that intends to have a neural network create songs of the late club of the 27 (artists that have tragically died at age 27 or near, and in the height of their respective careers), artists such as Amy Winehouse, Jimmy Hendrix, Curt Cobain and Jim Morrison.

The project was created by Over the Bridge, an organization dedicated to increase awareness on mental health and substance abuse in the music industry, trying to denormalize... read more

The omnipresence of technology nowadays has made it commonplace to read news about AI, just a quick glance at today’s headlines, and I get:

• This Powerful AI Technique Led to Clashes at Google and Fierce Debate in Tech.
• How A.I.-powered companies dodged the worst damage from COVID
• AI technology detects ‘ticking time bomb’ arteries
• AI in Drug Discovery Starts to Live Up to the Hype
• Pentagon seeks commercial solutions to get its data ready for AI

Topics from business, manufacturing, supply chain, medicine and biotech and even defense are covered in those news... read more

Background

After many years of a corporate career (17) diverging from computer science, I have now decided to learn Machine Learning and in the process return to coding (something I have always loved!).

To fully grasp the essence of ML I decided to start by coding a ML library myself, so I can fully understand the inner workings, linear algebra and calculus involved in Stochastic Gradient Descent. And on top learn Python (I used to code in C++ 20 years ago).

I built a general purpose basic ML library that... read more

Background

After many years of a corporate career (17) diverging from computer science, I have now decided to learn Machine Learning and in the process return to coding (something I have always loved!).

To fully grasp the essence of ML I decided to start by coding a ML library myself, so I can fully understand the inner workings, linear algebra and calculus involved in Stochastic Gradient Descent. And on top learn Python (I used to code in C++ 20 years ago).

I built a general purpose basic ML library that... read more

For the seemingly small project I undertook of creating a machine learning neural network that could learn by itself to play tic-tac-toe, I bumped into the necesity of implementing at least one momentum algorithm for the optimization of the network during backpropagation.

And since my original post for the TicTacToe project is quite large already, I decided to post separately these optimization methods and how did I implement them in my code.

Adam

source

Adaptive Moment Estimation (Adam) is an optimization method that computes adaptive learning rates for each weight and bias. In addition to storing an... read more

It must sound crazy that in this day and age, when we have such a myriad of amazing machine learning libraries and toolkits all open sourced, all quite well documented and easy to use, I decided to create my own ML library from scratch.

Let me try to explain; I am in the process of immersing myself into the world of Machine Learning, and to do so, I want to deeply understand the basic concepts and its foundations, and I think that there is no better way to do so than by creating myself all the... read more

Recently I was reading an article about a cool project that intends to have a neural network create songs of the late club of the 27 (artists that have tragically died at age 27 or near, and in the height of their respective careers), artists such as Amy Winehouse, Jimmy Hendrix, Curt Cobain and Jim Morrison.

The project was created by Over the Bridge, an organization dedicated to increase awareness on mental health and substance abuse in the music industry, trying to denormalize... read more

Recently I was reading an article about a cool project that intends to have a neural network create songs of the late club of the 27 (artists that have tragically died at age 27 or near, and in the height of their respective careers), artists such as Amy Winehouse, Jimmy Hendrix, Curt Cobain and Jim Morrison.

The project was created by Over the Bridge, an organization dedicated to increase awareness on mental health and substance abuse in the music industry, trying to denormalize... read more

For the seemingly small project I undertook of creating a machine learning neural network that could learn by itself to play tic-tac-toe, I bumped into the necesity of implementing at least one momentum algorithm for the optimization of the network during backpropagation.

And since my original post for the TicTacToe project is quite large already, I decided to post separately these optimization methods and how did I implement them in my code.

Adam

source

Adaptive Moment Estimation (Adam) is an optimization method that computes adaptive learning rates for each weight and bias. In addition to storing an... read more

In the midst of a global pandemic caused by the SARS-COV2 coronavirus; I decided to start blogging. I wanted to blog since a long time, I have always enjoyed writing, but many unknowns and having “no time” for it prevented me from taking it up. Things like: “I don’t really know who my target audience is”, “what would my topic or topics be?”, “I don’t think I am a world-class expert in anything”, and many more kept stopping me from setting up my own blog. Now seemed like a good time as any so with those and tons of other... read more

In the midst of a global pandemic caused by the SARS-COV2 coronavirus; I decided to start blogging. I wanted to blog since a long time, I have always enjoyed writing, but many unknowns and having “no time” for it prevented me from taking it up. Things like: “I don’t really know who my target audience is”, “what would my topic or topics be?”, “I don’t think I am a world-class expert in anything”, and many more kept stopping me from setting up my own blog. Now seemed like a good time as any so with those and tons of other... read more

Recently I was reading an article about a cool project that intends to have a neural network create songs of the late club of the 27 (artists that have tragically died at age 27 or near, and in the height of their respective careers), artists such as Amy Winehouse, Jimmy Hendrix, Curt Cobain and Jim Morrison.

The project was created by Over the Bridge, an organization dedicated to increase awareness on mental health and substance abuse in the music industry, trying to denormalize... read more

The omnipresence of technology nowadays has made it commonplace to read news about AI, just a quick glance at today’s headlines, and I get:

• This Powerful AI Technique Led to Clashes at Google and Fierce Debate in Tech.
• How A.I.-powered companies dodged the worst damage from COVID
• AI technology detects ‘ticking time bomb’ arteries
• AI in Drug Discovery Starts to Live Up to the Hype
• Pentagon seeks commercial solutions to get its data ready for AI

Topics from business, manufacturing, supply chain, medicine and biotech and even defense are covered in those news... read more

Background

After many years of a corporate career (17) diverging from computer science, I have now decided to learn Machine Learning and in the process return to coding (something I have always loved!).

To fully grasp the essence of ML I decided to start by coding a ML library myself, so I can fully understand the inner workings, linear algebra and calculus involved in Stochastic Gradient Descent. And on top learn Python (I used to code in C++ 20 years ago).

I built a general purpose basic ML library that... read more

Recently I was reading an article about a cool project that intends to have a neural network create songs of the late club of the 27 (artists that have tragically died at age 27 or near, and in the height of their respective careers), artists such as Amy Winehouse, Jimmy Hendrix, Curt Cobain and Jim Morrison.

The project was created by Over the Bridge, an organization dedicated to increase awareness on mental health and substance abuse in the music industry, trying to denormalize... read more

The omnipresence of technology nowadays has made it commonplace to read news about AI, just a quick glance at today’s headlines, and I get:

• This Powerful AI Technique Led to Clashes at Google and Fierce Debate in Tech.
• How A.I.-powered companies dodged the worst damage from COVID
• AI technology detects ‘ticking time bomb’ arteries
• AI in Drug Discovery Starts to Live Up to the Hype
• Pentagon seeks commercial solutions to get its data ready for AI

Topics from business, manufacturing, supply chain, medicine and biotech and even defense are covered in those news... read more

It must sound crazy that in this day and age, when we have such a myriad of amazing machine learning libraries and toolkits all open sourced, all quite well documented and easy to use, I decided to create my own ML library from scratch.

Let me try to explain; I am in the process of immersing myself into the world of Machine Learning, and to do so, I want to deeply understand the basic concepts and its foundations, and I think that there is no better way to do so than by creating myself all the... read more

For the seemingly small project I undertook of creating a machine learning neural network that could learn by itself to play tic-tac-toe, I bumped into the necesity of implementing at least one momentum algorithm for the optimization of the network during backpropagation.

And since my original post for the TicTacToe project is quite large already, I decided to post separately these optimization methods and how did I implement them in my code.

Adam

source

Adaptive Moment Estimation (Adam) is an optimization method that computes adaptive learning rates for each weight and bias. In addition to storing an... read more

Target Tracking Project

We designed image processing and control algorithms for a two-degree-of-freedom laser workbench equipped with a monocular camera, enabling stepper motor control for target identification and tracking.

Task specification

1. fitting inverse kinematics(optional and not done in video).
2. Scan the QR code to get a color sequence.
3. Use laser to point corresponding target both statically and dynamically.
4. Change next color with step 3.

6-DOF Robotic Arm Sorting Project

We developed forward and inverse kinematics and trajectory planning algorithms for a 6 dof robotic arm platform, achieving dart positioning and transportation.

Task specification

1. Taking a photo of the plane and get their position(not shown in video).
2. Mark half of the darts as the destination of the other half.(e.g. 3 origination 3 destination for 6 darts)
3. Place every dart to their assigned destination steadily and accurately.
4. Reset to the initial position.

Background

After many years of a corporate career (17) diverging from computer science, I have now decided to learn Machine Learning and in the process return to coding (something I have always loved!).

To fully grasp the essence of ML I decided to start by coding a ML library myself, so I can fully understand the inner workings, linear algebra and calculus involved in Stochastic Gradient Descent. And on top learn Python (I used to code in C++ 20 years ago).

I built a general purpose basic ML library that... read more

It must sound crazy that in this day and age, when we have such a myriad of amazing machine learning libraries and toolkits all open sourced, all quite well documented and easy to use, I decided to create my own ML library from scratch.

Let me try to explain; I am in the process of immersing myself into the world of Machine Learning, and to do so, I want to deeply understand the basic concepts and its foundations, and I think that there is no better way to do so than by creating myself all the... read more

I am lately trying to take on coding again. It had always been a part of my life since my early years when I learned to program a Tandy Color Computer at the age of 8, the good old days.

Tandy Color Computer TRS80 III

Having already programed in Java, C# and of course BASIC, I thought it would be a great idea to learn Python since I have great interest in data science and machine learning, and those two topics seem to have an avid community within Python coders.

For one of my starter quick programming... read more

Background

After many years of a corporate career (17) diverging from computer science, I have now decided to learn Machine Learning and in the process return to coding (something I have always loved!).

To fully grasp the essence of ML I decided to start by coding a ML library myself, so I can fully understand the inner workings, linear algebra and calculus involved in Stochastic Gradient Descent. And on top learn Python (I used to code in C++ 20 years ago).

I built a general purpose basic ML library that... read more

Statistics

This is note3 based on Linear Algebra And Learning From Data by Gilbert Strang.

1.Covariance Matrices and Joint Probabilities

Linear algebraenters When we run M different experiments. For example, we measure age and height and weight to get a vector $m=(m_1,m_2,m_3)$ in which $m=3$.

A matrix becomes involved when we look at variances. Each experiment (dimension) will have a sample variance or expected variance. If we measure age and height and weight for children, the results will be strongly correlated. The connection of variables is denoted by covariance.

\(\sigma_{12}=E[(m_1-\mu_1)(m_2-\mu_2)]\)
also: \(\sigma_{xy}=E[(X-\bar X)(Y-\bar Y)]\)
... read more

Statistics Note2

This is note2 based on MIT 18.650 .

1.Trinity of statistical inference

1.Estimation
2.Confidence intervals
3.Hypothesis testing

2.Statistical model

\((E,\rm (I\!P_\theta)_{\theta \in \Theta})\)
E: Sample space

$(P_\theta)_{\theta \in \Theta}$ is a family of probability

$\Theta$:parameter set

3.Estimation

Parameter Estimation

Statistic: Any measurable function of the sample, e.g.,
$\bar Xn, maxX_i, X_1 + log(1 + |Xn|), sample\ variance, etc..$

Estimator of $\theta$: Any statistic whose expression does not depend on $\theta$. (r.v.) \(\hat{\theta}_n\to\theta\)
Asymptotically normal:
\(\sqrt{n}(\hat{\theta}_n-\theta)\to N(0,\sigma^2)\)

Bias of an estimator

\(bias(\hat{\theta}_n)=E[\hat{\theta}_n]-\theta\) If bias =... read more

Statistics Note1

This is note1 based on Linear Algebra And Learning From Data by Gilbert Strang.

1.Big picture

When an ouput is predicted, we need its probability. When that output is measured,we need its statistics.

2.Mean,Variance

Sample mean : $m=\mu=\frac{1}{N}(x_{1}+x_{2}+\cdots+x_{N})$
Expected Value : $m=E[x]=p_{1}x_{1}+p_{2}x_{2}+\cdots+p_{n}x_{n}$

Law of Large Numbers: $\mu\to E[x]$ , as N increases.

Sample Variance:
$S^{2}=\frac{1}{N-1}[(x_{1}-m)^2+(x_{2}-m)^2+\cdots +(x_{N}-m)^2]$
Variance: $\sigma^2=E[(x-E[x])^2]=p_1(x_1-m)^2+p_2(x_2-m)^2+\cdots +p_n(x_n-m)^2$

The variance $\sigma ^2$ measures the expected distance(squared) from the expected mean E[x].
The sample variance measures actual distance(squared) form the actual sample mean $\mu$.

Notice!... read more

In the midst of a global pandemic caused by the SARS-COV2 coronavirus; I decided to start blogging. I wanted to blog since a long time, I have always enjoyed writing, but many unknowns and having “no time” for it prevented me from taking it up. Things like: “I don’t really know who my target audience is”, “what would my topic or topics be?”, “I don’t think I am a world-class expert in anything”, and many more kept stopping me from setting up my own blog. Now seemed like a good time as any so with those and tons of other... read more