help-octave
[Top][All Lists]

## Re: OT: finding the weights used in weighted least squares regression

 From: Mike Miller Subject: Re: OT: finding the weights used in weighted least squares regression Date: Thu, 28 Apr 2011 11:56:02 -0500 (CDT) User-agent: Alpine 2.00 (DEB 1167 2008-08-23)

```On Tue, 26 Apr 2011, Kamaraju S Kusumanchi wrote:

```
```The weighted least squares regression is done by solving
W A x = W b
for x, where W is nxn, A is nxm, x is mx1, b is nx1. W is a diagonal matrix.

```
My question is that, in the above set up, if we are given A, x, b is it possible to solve for the W?
```

We usually have an equation of this form:

y = X*b + e

```
The y vector (nx1) and X matrix (nxp) are given and we want to solve for b (px1) attempting to minimize the variance of hypothetical e (nx1) -- the least squares solution, but we assume that the variance-covariance matrix of e is of the form eye(n)*s^2 for some s. We can estimate s.
```
```
But what if the var-covar matrix of e is of a different form? Let's give it the general form V (nxn, not necessarily diagonal) where V is symmetric non-negative definite. It turns out that if V is positive definite (all positive eigenvalues), you can produce a matrix U=chol(inv(V)) such that
```
U*y = U*X*b + U*e

```
where U*e has var-covar matrix eye(n). This makes it possible to get a proper solution for b like so:
```
b = (U*X)\(U*y)

```
I assume that your "A x = b" is what I would call "X b = y". In other words, I think the W you want is the U I've given.
```
```
This means that you have to know something about the structure of your residuals. Are there groups of observations that allow you to estimate residual variances for the elements of e? If so, you can use an iterative feasible generalized least squares method:
```
(1) b = X\y ;   [implicitly assumes cov(e) = eye(n)*s^2]

(2) eOLS = y - X*b ;  [compute residuals]

```
(3) compute var-covar matrix V based on eOLS -- I can't tell you how because I don't know the structure of your data.
```
(4) U = chol(inv(V)) ;

(5) b = (U*X)\(U*y) ;

(6) eGLS = y - X*b ;

```
(7) compute var-covar matrix V based on eGLS -- again, I can't tell you how because I don't know the structure of your data.
```
```
(8) repeat steps 4-7 until b stops changing. It should not require many iterations.
```
Mike

--
Michael B. Miller, Ph.D.
Bioinformatics Specialist
Minnesota Center for Twin and Family Research
Department of Psychology
University of Minnesota

```