Least Square Regression Archives - OnlineEngineeringNotes https://onlineengineeringnotes.com/tag/least-square-regression/ A Complete Guide to future Engineers Wed, 26 May 2021 08:27:54 +0000 en-US hourly 1 https://wordpress.org/?v=6.5.2 Interpolation and Approximation: Numerical Differentiation and Regression https://onlineengineeringnotes.com/2021/05/26/interpolation-and-approximation-numerical-differentiation-and-regression/ https://onlineengineeringnotes.com/2021/05/26/interpolation-and-approximation-numerical-differentiation-and-regression/#respond Wed, 26 May 2021 08:27:50 +0000 https://onlineengineeringnotes.com/?p=568 INTERPOLATION Introduction The process of estimating intermediate values between given data points is called Interpolation . The most common method used for this purpose is Polynomial Interpolation. Find the functional value for π‘₯ = 2.5 , i.e. 𝑓 2.5 = ? Is this a problem of Interpolation or extrapolation?? π‘₯ 1 3 4 6 9 ... Read more

The post Interpolation and Approximation: Numerical Differentiation and Regression appeared first on OnlineEngineeringNotes.

]]>

INTERPOLATION

Introduction

  • The process of estimating intermediate values between given data points is called Interpolation .
  • The most common method used for this purpose is Polynomial Interpolation.

Find the functional value for π‘₯ = 2.5 , i.e. 𝑓 2.5 = ? Is this a problem of Interpolation or extrapolation??

π‘₯13469
𝑓(π‘₯)416436723

Solution:
Step 1: Obtain the polynomial equation using tabular data points.
Suppose we got our polynomial as,
𝑓 π‘₯ = 4π‘₯
4 + 3π‘₯
2 + 2
Step 2 : Now, put π‘₯ = 2.5 in the above expression of 𝑓 π‘₯ which gives the desired answer

Note: The number of data points minus one defines the order of interpolation. If n number of data points are given, the order of the polynomial will be n-1.

Linear Interpolation

  • Connecting two data points with a straight line.
  • Suppose we are given two data points (π‘₯1, 𝑓 π‘₯1 ) and (π‘₯2, 𝑓 π‘₯2 ) .
  • We are going to derive the formula which helps to calculate 𝑓 π‘₯ for the given π‘₯ where π‘₯1 ≀ π‘₯ ≀ π‘₯2 .

Let us find the equation of straight line from the given two points (π‘₯1, 𝑓 π‘₯1 ) and (π‘₯2, 𝑓 π‘₯2 ) .

Demerits of Linear Interpolation

  • Linear Interpolation uses first order polynomial (i.e. straight line) to approximate the function.
  • To reduce this error, we need to use higher order polynomial.

Lagrange Interpolation Polynomial

  • Let π‘₯0, 𝑓0 , π‘₯1, 𝑓1 , and π‘₯2, 𝑓2 are the given data points.
xπ‘₯0π‘₯1π‘₯2
f𝑓0𝑓1𝑓2

Here, number of data points = 3 , so the polynomial will be of the order 2.

  • Let us consider a second order polynomial of the form 𝑝2 π‘₯ = 𝑏1 π‘₯ βˆ’ π‘₯0 π‘₯ βˆ’ π‘₯1 + 𝑏2 π‘₯ βˆ’ π‘₯1 π‘₯ βˆ’ π‘₯2 + 𝑏3 π‘₯ βˆ’ π‘₯2 π‘₯ βˆ’ π‘₯0 ………………..[1] where 𝑏1, 𝑏2 π‘Žπ‘›π‘‘ 𝑏3 π‘Žπ‘Ÿπ‘’ π‘‘β„Žπ‘’ π‘π‘œπ‘“π‘“π‘–π‘π‘–π‘’π‘›π‘‘π‘  π‘œπ‘“ π‘π‘œπ‘™π‘¦π‘›π‘œπ‘šπ‘–π‘Žπ‘™.
  • Given points π‘₯0, 𝑓0 , π‘₯1, 𝑓1 , and π‘₯2, 𝑓2 should satisfy the equation 1.
  • At π‘₯ = π‘₯π‘œ, 𝑝2 π‘₯0 = π‘“π‘œ,

Similarly, for π‘₯ = π‘₯1, 𝑝2 π‘₯1 = 𝑓1 Β 

Example 2.1
For the following data ,use Lagrange interpolation polynomial to find 𝑦 π‘Žπ‘‘ π‘₯ = 3.2.

x1245
y4961120

Solution: Here, π‘₯0 = 1, π‘₯1 = 2, π‘₯2 = 4, π‘₯3 = 5 , 𝑓0 = 4, 𝑓1 = 9, 𝑓2 = 61, 𝑓3 = 120

Now,

Newton Interpolation Polynomial

  • The Newton form of polynomial is 𝒑𝒏 (𝒙) = π’‚πŸŽ+ π’‚πŸπ’™ βˆ’ π’™πŸŽ + π’‚πŸ 𝒙 βˆ’ π’™πŸŽ 𝒙 βˆ’ π’™πŸ + π’‚πŸ‘ 𝒙 βˆ’ π’™πŸŽ 𝒙 βˆ’ π’™πŸ 𝒙 βˆ’ π’™πŸβ€¦ … … … … . . +𝒂𝒏 𝒙 βˆ’ π’™πŸŽ 𝒙 βˆ’ π’™πŸ … . . (𝒙 βˆ’ π’™π’βˆ’πŸ)
  • For given π‘₯0, 𝑓0 , π‘₯1, 𝑓1 , π‘₯2, 𝑓2 … … … (π‘₯𝑛, 𝑓𝑛), we need to find the value of coefficients π‘Ž0, π‘Ž1, π‘Ž2, π‘Ž3 … … … . . π‘Žπ‘›.
βœ“ If only two data points π‘₯0, 𝑓0 , π‘₯1, 𝑓1 are given, π’‘πŸ 𝒙 = π’‚πŸŽ+ π’‚πŸ 𝒙 βˆ’ π’™πŸŽ β†’ πŸπ’”π’• 𝒐𝒓𝒅𝒆𝒓 π’‘π’π’π’šπ’π’π’Žπ’Šπ’‚π’
βœ“ If three data points π‘₯0, 𝑓0 , π‘₯1, 𝑓1 , π‘₯2, 𝑓2 are given π’‘πŸ 𝒙 = π’‚πŸŽ + π’‚πŸ 𝒙 βˆ’ π’™πŸŽ + π’‚πŸ 𝒙 βˆ’ π’™πŸŽ 𝒙 βˆ’ π’™πŸ β†’ πŸπ’π’… 𝒐𝒓𝒅𝒆𝒓 π’‘π’π’π’šπ’π’π’Žπ’Šπ’‚π’
βœ“ Similarly, for four data points π‘₯0, 𝑓0 , π‘₯1, 𝑓1 , π‘₯2, 𝑓2 π‘₯3, 𝑓3 , π’‘πŸ‘ 𝒙 = π’‚πŸŽ + π’‚πŸ 𝒙 βˆ’ π’™πŸŽ + π’‚πŸ 𝒙 βˆ’ π’™πŸŽ 𝒙 βˆ’ π’™πŸ + π’‚πŸ‘(𝒙 βˆ’ π’™πŸŽ)(𝒙 βˆ’ π’™πŸ)(𝒙 βˆ’ π’™πŸ) β†’ πŸ‘π’“π’… 𝒐𝒓𝒅𝒆𝒓 π’‘π’π’π’šπ’π’π’Žπ’Šπ’‚π’
  • For π‘₯ = π‘₯0, 𝑝𝑛 (π‘₯0 )= 𝑓0 , we will get 𝑓0 = π‘Ž0 . π‘Ž0 = 𝑓0
  • For π‘₯ = π‘₯1, 𝑝𝑛 (π‘₯1 ) = 𝑓

or,  π‘“1 = π‘Ž0 + π‘Ž1 π‘₯1 βˆ’ π‘₯0

or, 𝑓1 = 𝑓0 + π‘Ž1 π‘₯1 βˆ’ π‘₯0 [since π‘Ž0 = 𝑓0]

𝑓2 = π‘Ž0 + π‘Ž1 π‘₯2 βˆ’ π‘₯0 + π‘Ž2(π‘₯2 βˆ’ π‘₯0)(π‘₯2 βˆ’ π‘₯1) which gives

Newton Divide Difference Table

The alternative way of finding the coefficients (π‘Ž0, π‘Ž1, π‘Ž2 π‘Žπ‘›π‘‘ π‘ π‘œ π‘œπ‘›) value is to use Newton divided difference table. For given π‘₯0, 𝑓0 , π‘₯1, 𝑓1 , π‘₯2, 𝑓2 , π‘₯3, 𝑓3 π‘Žπ‘›π‘‘ (π‘₯4, 𝑓4)

Example 2.2

Find the functional value for π‘₯ = 7 using Newton interpolation polynomial.

x56911
𝑓(π‘₯)12131416

Solution: Since 4 data points are given, the required polynomial will be of the order 3.

 π’‘πŸ‘ (𝒙) = π’‚πŸŽ + π’‚πŸ 𝒙 βˆ’ π’™πŸŽ + π’‚πŸ 𝒙 βˆ’ π’™πŸŽ 𝒙 βˆ’ π’™πŸ + π’‚πŸ‘ 𝒙 βˆ’ π’™πŸŽπ’™ βˆ’ π’™πŸ 𝒙 βˆ’ π’™πŸ

To get the value of π‘Ž0, π‘Ž1, π‘Ž2, π‘Ž3 π‘Žπ‘›π‘‘ π‘Ž4 , we are going to use Newton divided difference table.

Newton Forward Difference /Newton Backward Difference [Interpolation with equidistant points]

  • Particularly used when the functional values are given at a sequence of equally spaced points.
  • Also called as Newton Gregory technique.

Scenario 1:

x56911
𝑓(π‘₯)12131416

π»π‘’π‘Ÿπ‘’, π‘₯1 βˆ’ π‘₯0 = 6 βˆ’ 5 = 1 π‘€β„Žπ‘’π‘Ÿπ‘’π‘Žπ‘ , π‘₯2 βˆ’ π‘₯1 = 9 βˆ’ 6 = 3 Not equally spaced. We can’t use Newton Forward/Backward Difference. Rather, we need to rely on Lagrange or Newton Interpolation polynomial.

Scenario 2:

x1020304050
𝑓(π‘₯)072663124

Here, 𝒙 values are equally spaced. In this case we can use Newton forward difference or Newton backward difference.
𝑠𝑑𝑒𝑝 𝑠𝑖𝑧𝑒 β„Ž = 10

  • If required point is close to the start of the table, use Newton forward difference. 𝐴𝑑 π‘₯ = 15, 𝑓 (π‘₯) = ? π‘œπ‘Ÿ 𝐴𝑑 π‘₯ = 25, 𝑓 π‘₯ = ? [ First value of 𝒙 (𝐒. 𝐞. π‘₯0) acts as a reference point ]
  • If required point is close to the end of the table , use Newton backward difference. 𝐴𝑑 π‘₯ = 48, 𝑓 π‘₯ =? π‘œπ‘Ÿ 𝐴𝑑 π‘₯ = 35, π‘₯ = ? [ Last value of 𝒙 (𝐒. 𝐞. π‘₯𝑛) acts as a reference point ]
  • Simple difference is used here rather than divided difference which was used in Newton divided difference table. 𝑓1 βˆ’ 𝑓0 = 1𝑠𝑑 π‘œπ‘Ÿπ‘‘π‘’π‘Ÿ π‘ π‘–π‘šπ‘π‘™π‘’ π‘‘π‘–π‘“π‘“π‘’π‘Ÿπ‘’π‘›π‘π‘’

Newton Forward Difference

Let us create a forward difference table for given π‘₯0, 𝑓0 , π‘₯1, 𝑓1 , π‘₯2, 𝑓2 , π‘₯3, 𝑓3 π‘Žπ‘›π‘‘ (π‘₯4, 𝑓4)

Then, the Newton forward difference formula is given by,

Example 2.3:

Estimate the value of π‘ π‘–π‘›πœƒ π‘Žπ‘‘ πœƒ = 250 using Newton-Gregory forward difference formula.

πœƒ1020304050
sin(πœƒ)0.17360.34200.50000.64280.7660

Solution: Let us draw a Newton forward difference table to find the value of coefficients

π’‡πŸŽ = 𝟎. πŸπŸ•πŸ‘πŸ” , βˆ†π’‡πŸŽ = 𝟎. πŸπŸ”πŸ–πŸ’, βˆ† πŸπ’‡πŸŽ = βˆ’πŸŽ. πŸŽπŸπŸŽπŸ’ , βˆ† πŸ‘π’‡πŸŽ = βˆ’πŸŽ. πŸŽπŸŽπŸ’πŸ–, βˆ† πŸ’π’‡πŸŽ = 𝟎. 𝟎𝟎𝟎4

Newton forward difference formula is given by,

= 0.422665625

Newton Backward Difference

Let us create a forward difference table for given π‘₯0, 𝑓0 , π‘₯1, 𝑓1 , π‘₯2, 𝑓2 , π‘₯3, 𝑓3 π‘Žπ‘›π‘‘ (π‘₯4, 𝑓4)

Then, the Newton backward difference formula is given by,

Example 2.4
Estimate the value of π‘ π‘–π‘›πœƒ π‘Žπ‘‘ πœƒ = 45 0using Newton-Gregory backward difference formula.

πœƒ1020304050
sin(πœƒ)0.17360.34200.50000.64280.7660

Solution: Let us draw a Newton backward difference table to find the value of coefficients

From table, 𝑓4 = 0. 7660, 𝛻𝑓4 = 0.1232, 𝛻 2 𝑓4 = βˆ’0. 0196, 𝛻 3 𝑓4 = βˆ’0.0044, π‘Žπ‘›π‘‘ 𝛻 4 𝑓4 = 0.0004

Newton backward difference formula is given by

= 0.707509

NUMERICAL DIFFERENTIATION

Numerical Differentiation

  • Numerical differentiation is the process of computing the value of the derivative of an explicitly unknown function, with given discrete set of points(π‘₯𝑖 , 𝑓𝑖) .
  • To differentiate a function numerically, we first determine an interpolating polynomial and then compute the approximate derivative at the given point.
  • If π‘₯𝑖 ’s are equispaced use Newton’s forward/backward interpolation formula to find the derivative.
  • If π‘₯𝑖 ’s are not equispaced, we may find using Newton’s divided difference method or Lagrange’s interpolation formula and then differentiate it as many times as required.
Problem Scenario:
The following table gives the displacement, x (cm) of an object at various of time ,t(seconds). Find the velocity and acceleration of the object at t=1.2 sec. Use suitable interpolation method.
t 1.0 1.2 1.4 1.6 1.8
x 9.0 9.5 10.2 11.0 13.2

Numerical Derivatives using Newton Forward Interpolation

Newton forward interpolation formula is given by,

Numerical Derivatives using Newton Backward Interpolation

For given π‘₯0, 𝑓0 , π‘₯1, 𝑓1 , π‘₯2, 𝑓2 , π‘₯3, 𝑓3 π‘Žπ‘›π‘‘ (π‘₯4, 𝑓4) , Newton backward interpolation formula is given by

Example 2.5

x0123
y5638

Solution:

Newton difference table,

Formula for Newton forward interpolation is given by,

Example 2.6 Following table gives the census population of a state for the 1961 to 2001.

Year19611971198119912001
Population(million)19.9636.6558.8177.2194.61

Find the rate of growth of the population in the year 2001.
Solution: since given table is equi-spaced (h=10) and rate of growth is asked for 2001, we have to
use Newton backward interpolation.

𝑓4 = 94.61, βˆ‡ 𝑓4 = 17.40, βˆ‡ 2 𝑓4 = βˆ’1, βˆ‡ 3 𝑓4 = 2.76, βˆ‡ 4 𝑓4 = 11.99

Newton Backward Interpolation formula is give by,

Example 2.7 Compute f ’(3.5) and f ’’(4) given that f(0)=2, f(1)=3, f(2)=12, and f(5)=147

Solution:

x0125
y2312147

Since π‘₯𝑖 β€²s values are not equally spaced, we have to use either Lagrange or Newton divided difference formula.

Let us draw Newton divide difference table,

From table, π‘Ž0 = 2, π‘Ž1 = 1, π‘Ž2 = 4, π‘Ž3 = 1

Newton interpolation formula is given by,

 π‘“( π‘₯) = 𝑝3( π‘₯ )= π‘Ž0 + π‘Ž1 π‘₯ βˆ’ π‘₯0 + π‘Ž2 π‘₯ βˆ’ π‘₯0 π‘₯ βˆ’ π‘₯1 + π‘Ž3(π‘₯ βˆ’ π‘₯0)(π‘₯ βˆ’ π‘₯1)(π‘₯ βˆ’ π‘₯2)

Substituting the value of coefficients,

𝑓( π‘₯) = 2 + 1 π‘₯ βˆ’ 0 + 4 π‘₯ βˆ’ 0 π‘₯ βˆ’ 1 + 1(π‘₯ βˆ’ 0)(π‘₯ βˆ’ 1)(π‘₯ βˆ’ 2)

 𝑓 (π‘₯) = π‘₯ 3 + π‘₯ 2 βˆ’ π‘₯ + 2

REGRESSION

Regression : Introduction

β€’Interpolation: If we’re provided with various scattered data points then we try to find a curve
such that all the data points will exactly pass through it.(Perfect match)
β€’Regression: Here we try to fit a specific form of curve to the given data points. So, it may be
possible that all the points might not pass through the curve.(Generalizes the data)

Linear Regression

Fitting straight line to the given data points.

For given data points we need only one straight line (linear) which best fits the data.
The general form of equation:
π’š = 𝒂 + 𝒃𝒙
Value of coefficients a, b will be calculated with the help of given data points.

Linear Regression : Best Fitting Techniques

π‘’π‘Ÿπ‘Ÿπ‘œπ‘Ÿ = π‘‘π‘Ÿπ‘’π‘’ π‘£π‘Žπ‘™π‘’π‘’ βˆ’ π‘£π‘Žπ‘™π‘’π‘’ π‘œπ‘π‘‘π‘Žπ‘–π‘›π‘’π‘‘ π‘“π‘Ÿπ‘œπ‘š 𝑓𝑖𝑑𝑑𝑒𝑑 π‘’π‘žπ‘’π‘Žπ‘‘π‘–π‘œπ‘›

 π‘’0 = 𝑦0 βˆ’ (π‘Ž + 𝑏π‘₯0)

𝑒1 = 𝑦1 βˆ’ (π‘Ž + 𝑏π‘₯1)

𝑒2 = 𝑦2 βˆ’ (π‘Ž + 𝑏π‘₯2)

 π‘’3 = 𝑦3 βˆ’ (π‘Ž + 𝑏π‘₯3) 𝑦 = π‘Ž + 𝑏π‘₯

Β In General, 𝑒𝑖 = 𝑦𝑖 βˆ’ π‘Ž + 𝑏π‘₯𝑖 = 𝑦𝑖 βˆ’ π‘Ž βˆ’ 𝑏π‘₯i

Least Square Regression
I. Fitting Linear Equation of the form π’š = 𝒂 + 𝒃x

Error for each data point can be expressed as ,

𝑒𝑖 = 𝑦𝑖 βˆ’ π‘Ž + 𝑏π‘₯𝑖 = 𝑦𝑖 βˆ’ π‘Ž – 𝑏π‘₯𝑖

 In least squared approach, we use sum of square of errors, i.e

Total Error 𝑄 = 𝑒1 2 + 𝑒2 2 + 𝑒3 2 + 𝑒4 2 + … … … . . +𝑒𝑛 2

We have to choose π‘Ž π‘Žπ‘›π‘‘ 𝑏 such that total error 𝑄 is minimum. Since 𝑄 depends on π‘Ž π‘Žπ‘›π‘‘ 𝑏 , a necessary
condition for 𝑄 to be minimum is,

Example 2.8 Fit a straight line for the following data

x12345
y34568

Solution:

We have to fit straight line of the form
𝑦 = π‘Ž + 𝑏π‘₯
Normal equations for the above equation is,

n = 5 (number of data points given in table)
Now,
26 = 5π‘Ž + 15𝑏
90 = 15π‘Ž + 55𝑏
Upon solving these two equations, we get
π‘Ž = 1.60, 𝑏 = 1.20
Hence, the required straight line is, 𝑦 = π‘Ž + 𝑏π‘₯ = 𝟏. πŸ” + 𝟏. 𝟐x

Example 2.9: If P is pull required to lift a load W by means of a pulley, find a linear law of the form 𝑃 =
π‘šπ‘Š + 𝑐 using the following data. Also, find the value of P for W =30.
P 12 15 21 25
W 50 70 100 120

Hint: In 𝑃 = π‘šπ‘Š + 𝑐, π‘š π‘Žπ‘›π‘‘ 𝑐 are the coefficients whose value is to be determined with the help of given
data.

II. Fitting Power Equation of the form π’š = 𝒂𝒃 𝒙

 The power equation 𝑦 = π‘Žπ‘ π‘₯ is non linear in the form. First of all, we need to represent it into linear form.

 π‘¦ = π‘Žπ‘₯ 𝑏

 Taking 𝑙𝑛 on both sides,

ln 𝑦 = ln(π‘Žπ‘₯ 𝑏 )

ln 𝑦 = ln π‘Ž + 𝑏 ln(π‘₯)

Β Let us suppose, π‘Œ = ln 𝑦 , 𝐴 = ln 𝐴 , 𝑋 = ln π‘₯ . We get, π‘Œ = 𝐴 + 𝑏 𝑋 (Linear Form )

References :

  1. Balagurusamy, E. Numerical Methods. New Delhi: Tata McGraw-Hill Publishing Company Limited, 2008.
  2. Sastry, S.S. Introductory Methods of Numerical Analysis. New Delhi: PHI Learning Pvt. Ltd., 2012.
  3. Steven C. Chopra, Raymond P. Canale. Numerical Methods for Engineers. Newyork: MCGraw Hill Education,
    2015.

The post Interpolation and Approximation: Numerical Differentiation and Regression appeared first on OnlineEngineeringNotes.

]]>
https://onlineengineeringnotes.com/2021/05/26/interpolation-and-approximation-numerical-differentiation-and-regression/feed/ 0