DE WIT’S METHOD

da | Ott 18, 2026 | Astronomy, Mathematics, Navigation | 0 commenti

The least-squares fix from several lines of position: theory, formulae, worked example and spreadsheet

In brief. Whenever more than two lines of position are available they never meet at a single point: they enclose a polygon, and for a century the choice of a point inside it was a matter of judgement. In 1974 Cornelis de Wit, of Delft University of Technology, gave that choice an objective rule: the position is the point that makes the sum of the squares of its distances from the lines a minimum. The rule is solved with five sums and two divisions, needs neither tables nor plotting, yields a measure of how good the fix is, and lends itself to a spreadsheet. It is the algorithm printed today on pages 282-283 of the Nautical Almanac and built into almost every celestial calculator on the market.

Source tags: [B] bibliographic source or official document · [C] the author’s own computation, verified numerically · [A] the author’s own assessment.

1.  The problem: three lines, three intersections

Every line of position is a locus: the vessel lies somewhere on that line. Two lines give a point, and the problem looks closed. But two lines allow no check at all: if one of them is wrong, the fix is wrong and nobody notices. This is why the prudent navigator takes three, four or five.

And this is where the trouble starts. Three lines never pass through one point: they enclose a triangle, the cocked hat. Four lines give a quadrilateral with six intersections, five lines give ten. The true position lies somewhere inside — but where exactly? Traditional practice offers three answers: the visual centre of the polygon, the meeting point of the bisectors when a systematic error is suspected, or the vertex that is most prudent with respect to dangers. All three are reasonable; none of the three is uniquely defined, and two navigators working on the same polygon will mark two different points.

De Wit’s method answers the same question with a definition that leaves no room for taste: the fix is the point that makes the sum of the squares of its distances from the observed lines a minimum. Once that definition is fixed, the point is computed — not chosen.

2.  Who de Wit was, and where his method lives today

Cornelis de Wit, of Delft University of Technology, published in the winter of 1974-75, in Navigation, the journal of the Institute of Navigation, the paper « Optimal Estimation of a Multi-Star Fix ». The idea was to apply to the celestial fix the least-squares estimation theory already familiar in geodesy, working not on the circles of equal altitude but on the straight lines that approximate them near the assumed position: a direct mathematical translation of what the navigator does on the chart. [B]

The same algorithm was developed independently at the Royal Greenwich Observatory and found its way into « Compact Data for Navigation and Astronomy » by Yallop and Hohenkerk; from there it passed into the Nautical Almanac, which prints it in the sight reduction section (pages 282-283 of the commercial edition) under the name « many body fix », with the instruction to repeat the computation for four iterations. Anyone using a celestial navigation program today, or the fix function of a dedicated calculator, is almost always using these formulae, often without knowing it. [B]

In 1992 de Wit returned to the subject in The Journal of Navigation with « Optimal Position Estimate from a Star’s Fix with Correlated Errors ». The new point matters to the practising navigator: instead of trying to separate the systematic error beforehand — index error, personal error, a badly assessed dip — it is left in the data and handled in the covariance matrix of the measurement errors. The result is an estimate that is free of bias and has the smallest possible spread. The practical translation of that idea is the three-unknown model of section 10. [B]

Both original papers are available only on payment from their publishers. What follows sets out the algorithm in the form in which it is publicly printed by the Nautical Almanac and in the form that follows directly from least-squares theory; the numerical verifications are the author’s own. [B] [C]

3.  The model: the line of position as a first-degree equation

Take an assumed position AP, of known coordinates, and build around it a plane tangent to the Earth. In this plane only two quantities are measured, both in minutes of arc, that is in nautical miles:

  • x  —  displacement to the North of the assumed position;
  • y  —  displacement to the East of the assumed position (the departure).

The intercept p of each observation is the difference between the observed altitude and the altitude computed for the assumed position, p = Ho − Hc, in minutes, reckoned positive towards the body. The computed azimuth is Zn. In the tangent plane the line of position is then the line perpendicular to the azimuth direction and at distance p from the assumed position:

x · cos Zn  +  y · sin Zn  =  p

This is the only approximation in the method, and it is the same one made whenever a straight line is drawn on the chart instead of an arc of a circle. It holds as long as the true position is not too far from the assumed one: within some twenty miles the linearisation error is negligible, within some sixty it is removed by a second pass (section 8). [A]

The line of position in the tangent plane: the azimuth gives the direction, the intercept the distance.

4.  The criterion: least squares

If the point sought is (x, y) and the i-th line has azimuth Zn(i) and intercept p(i), the quantity

r(i)  =  p(i)  −  [ x · cos Zn(i)  +  y · sin Zn(i) ]

is the signed distance between the point and that line, measured perpendicular to the line itself. If the point were perfect and the observations free of error, every r(i) would be zero. In practice they never are, and de Wit’s criterion selects the point that makes the sum of their squares a minimum:

Σ r(i)²  =  minimum

The square is not a mathematical whim. Squaring removes the signs — a line passing to the left and one passing to the right do not cancel each other — and penalises one badly discordant line more than two slightly discordant ones, which is exactly what is wanted of an averaging rule. Furthermore, if the observational errors are random and normally distributed, a reasonable assumption for a sextant in a steady hand, the resulting point is also the most probable of all. [B]

The heavy segments are the residuals: the method makes the sum of their squares a minimum.

Setting the derivatives of the sum with respect to x and to y equal to zero gives two linear equations in the two unknowns, the normal equations:

A · x  +  B · y  =  D
B · x  +  C · y  =  E

where the five coefficients are five sums taken over all the observed lines:

A = Σ cos² Zn        B = Σ sin Zn · cos Zn        C = Σ sin² Zn
D =
Σ p · cos Zn        E = Σ p · sin Zn

The system is solved by Cramer’s rule. With G = A·C − B²:

x = ( C·D − B·E ) / G          y = ( A·E − B·D ) / G

These are all the formulae of the method. There are no others: five sums, one subtraction and two divisions, whatever the number of lines.

5.  The working procedure

The seven steps. For each observation: 1) reduce the sight as usual, obtaining Hc and Zn from the assumed position, and compute the intercept p = Ho − Hc in minutes; 2) if the vessel is under way, transport the line to the time of the fix (section 9). Then, once only: 3) fill in the five columns of products and add them to obtain A, B, C, D, E; 4) compute G = A·C − B²; 5) obtain x and y; 6) apply them to the assumed position, remembering that y must be divided by the cosine of the latitude before it becomes a difference of longitude; 7) compute the residuals and the standard deviation, which tell whether the fix deserves confidence.

The step from x and y to the coordinates is the usual one, and it is the only place where latitude enters:

Lat = Lat(AP) + x          Long = Long(AP) + y / cos Lat

with x and y in minutes; the latitude to be used in the denominator is that of the assumed position (the difference from the mid-latitude is of second order and has no practical effect). [A]

6.  A complete worked example

Evening twilight of 15 March 2026, western Mediterranean south of Sardinia. The vessel is under way with course over ground (COG) 255° and speed over ground (SOG) 6.0 knots. Five stars are observed six minutes apart, from 17:42 to 18:06 UTC; the time of the fix is that of the last sight. The assumed position for that time is 39° 28.0′ N — 009° 22.0′ E. [C]

BodyUTCGHADecHo (observed, corrected)
Dubhe17:42:00272° 29.5′N 61° 36.5′39° 13.4′
Regulus17:48:00287° 53.8′N 11° 50.2′28° 19.7′
Sirius17:54:00340° 16.0′S 16° 44.8′33° 05.2′
Menkar18:00:00037° 26.6′N 04° 11.5′35° 14.2′
Mirfak18:06:00033° 18.6′N 49° 57.4′58° 31.1′

The altitudes Ho are already corrected for dip, refraction and index error. The ephemerides are computed, not copied from the Nautical Almanac: they may differ by a few tenths of a minute from the tabulated values, which in no way affects the demonstration of the method. [C]

Reduction of the five sights from the assumed position, with the transport to the time of the fix (d is the run remaining to that time, T its projection on the azimuth):

BodyHc (computed)Znp (‘)d (‘)T (‘)p’ (‘)
Dubhe39° 23.5′37.0°-10.12.4-1.9-12.0
Regulus28° 27.4′98.3°-7.71.8-1.7-9.3
Sirius32° 57.6′168.1°+7.71.2+0.1+7.7
Menkar34° 59.4′242.6°+14.90.6+0.6+15.5
Mirfak58° 24.2′303.7°+6.80.0+0.0+6.8

Now fill in the five columns of products. This is the heart of the method, and any pocket calculator will do it:

Bodycos Znsin Zncos² Znsin Zn cos Znsin² Znp’ cos Znp’ sin Zn
Dubhe+0.7984+0.60210.6374+0.48070.3626-9.549-7.202
Regulus-0.1440+0.98960.0207-0.14250.9793+1.346-9.249
Sirius-0.9787+0.20540.9578-0.20100.0422-7.564+1.587
Menkar-0.4607-0.88750.2123+0.40890.7877-7.123-13.722
Mirfak+0.5542-0.83240.3071-0.46130.6929+3.793-5.697
Sums  A = 2.1354B = +0.0849C = 2.8646D = -19.098E = -34.283

From here on it is arithmetic:

G = A·C − B² = 2.1354 × 2.8646 − (+0.0849)² = 6.1099
x = ( C·D − B·E ) / G = -8.48′          y = ( A·E − B·D ) / G = -11.72′

The fix therefore lies -8.48′ in latitude and -11.72′ of departure from the assumed position. To turn the departure into a difference of longitude, divide by the cosine of the latitude: -11.72′ / cos 39.5° = -15.18′. Hence:

Lat   =  39° 28.0′ N  −  8.48′   =  39° 19.5′ N
Long  =  009° 22.0′ E  −  15.18′  =  009° 06.8′ E

The true position of the simulation, which the navigator of course does not know, was 39° 19.4′ N — 009° 07.0′ E. The computed fix falls 0.20 miles from it, although it started from an assumed position more than fourteen miles out and although every observation carries a systematic error of +1.5′. [C]

Left, the five transported lines and the assumed position; right, the enlargement on the polygon, with the computed fix and the true position.

7.  How good is the computed fix

The method returns not only a position but also the means of judging it. The first is the residual of each line, that is the distance between the computed fix and the line itself, and the second is the standard deviation of all the lines:

σ  =  √ [ Σ r(i)²  /  ( n − 2 ) ]

The divisor is n − 2 and not n because two degrees of freedom have been used up in determining x and y: with only two lines σ cannot be computed, and indeed with two lines there is no check at all. In the example:

Bodyresidual r (‘)Check
Dubhe+1.86within 3 σ
Regulus+1.03within 3 σ
Sirius+1.84within 3 σ
Menkar+1.16within 3 σ
Mirfak+1.79within 3 σ

σ = 2.04′

Two minutes is a high value for five stars taken carefully from a steady deck, and the reason can be read in the column of residuals: they are all positive, and of similar size. Residuals all of the same sign are not chance, they are the signature of a systematic error — which is precisely the case here, where the systematic error is present and amounts to +1.5′. Section 10 shows how to deal with it.

A practical rule for reading it. σ below 1′ is an excellent set of sights; between 1′ and 2′ is normal at sea; between 2′ and 4′ there is almost always a systematic error or a bad line; above 4′ the fix should not be used before the reason has been understood. [A]

The geometric quality G

The denominator G = A·C − B² is not merely a step in the computation: it measures how well the azimuths are spread. Its largest value, with n lines perfectly distributed round the horizon, is n²/4; the further G falls below that value, the more sensitive the solution becomes to observational error. In the example G = 6.1099 against a theoretical maximum of 6.25: almost ideal geometry, because the five stars cover nearly the whole horizon.

If instead the azimuths are close together, G collapses and precision with it, however good the sextant. It is the same reason why two lines crossing at a narrow angle give an uncertain intersection on the chart — here, though, the fact is read in a number, before anything is plotted.

For the same quality of observation, azimuth geometry decides the size of the error ellipse.

The error ellipse

With σ and the coefficients already computed, the covariance matrix of the fix follows, that is the complete description of its uncertainty:

var(x) = σ² · C / G        var(y) = σ² · A / G        cov(x, y) = − σ² · B / G

The eigenvalues of this matrix are the squares of the semi-axes of the error ellipse, and the major eigenvector gives its orientation. In the example the 1 σ semi-axes are 1.40′ and 1.20′, with the major axis oriented 173°: the uncertainty is nearly circular, which is another way of saying that the geometry is good.

Multiplying the semi-axes by 2.45 gives the ellipse within which the true position falls with 95 % probability — but only if σ were known beforehand. When σ is estimated from few observations, as here, the correct factor is larger: with five lines it is about 4.4. The 95 % ellipse then measures 6.1′ by 5.2′ instead of 3.42′ by 2.94′. The difference matters when navigating close to a danger, and it is only honest to state it. [C]

The 1 σ and the 95 % error ellipses, with the true position of the simulation.

8.  Why the Nautical Almanac repeats the computation four times

The line of position is a straight line only by approximation: it is in reality an arc of a circle whose radius is the zenith distance. As long as the computed fix stays near the assumed position the difference is invisible; when it is far away, it is not. The remedy is elementary: the whole computation is repeated using the point just obtained as the new assumed position. The new Hc and Zn are then computed from the right place, and the result improves.

By how much? In the example, with the assumed position fourteen miles out, the second pass moves the fix by 0.02′ — that is, by nothing. Repeating the trial with an assumed position 57 miles out, the first pass lands 0.46 miles from the truth and the second 0.19: a gain of three tenths of a mile, and from the third pass on nothing changes at all. [C]

In practice. One pass is enough when the assumed position is within some twenty miles. Beyond that, a second one is made. The four iterations prescribed by the Almanac are there to guarantee convergence in every possible case, including the navigator who starts hundreds of miles out, and they cost a spreadsheet nothing. [A]

9.  The vessel under way

The five stars of the example were observed over twenty-four minutes, during which the vessel ran 2.4 miles. To ignore this would be to compute the fix of a stationary vessel that does not exist. The correct treatment is the usual one — transporting the line — but written numerically instead of graphically.

A time of fix is chosen, as a rule that of the last sight, together with a single assumed position referred to that time. For each line the remaining run d from the instant of the sight to the time of the fix is computed, and its projection on the azimuth direction is taken:

d = SOG × Δt / 60          T = d · cos ( Zn − COG )          p’ = p + T

with Δt in minutes of time and d in miles. The transported intercept p’ takes the place of p in all the formulae: the rest of the computation does not change by one comma. The sign looks after itself — if the vessel closes the body the term is positive, if it opens it is negative — and there is no rule to remember.

Transport: only the component of the run along the azimuth displaces the line.

In the example the transport amounts to 1.9′ at most for the first star and to zero for the last, which is the time of the fix. Two miles are a great deal if a fix to a tenth of a mile is wanted. [C]

10.  Systematic error and the third unknown

Section 7 showed that in the example all five residuals are positive. When this happens, the assumption of purely random errors is contradicted by the data themselves: some quantity is being added to every altitude in the same way. The causes are well known — an index error not measured or changed since, a dip different from the tabulated one because of anomalous refraction, a constant personal error of the observer, an indistinct horizon at twilight — and they all have the same consequence: every line is displaced by the same amount, in the same sense with respect to the body.

De Wit’s answer, in the 1992 paper, is not to try to remove this error before the computation but to include it in the computation as a third unknown. Each line is then written:

x · cos Zn(i)  +  y · sin Zn(i)  +  k  =  p'(i)

where k is the unknown systematic error, common to all the observations. The unknowns become three, the normal equations become a system of three equations in three unknowns — still soluble by Cramer’s rule, on 3 × 3 determinants — and at least four lines are needed for the problem to make sense (with three lines there would be three equations and three unknowns: an exact solution and no check whatever).

Applied to the example, the three-unknown model gives:

x = -8.31′        y = -11.76′        k = +1.54′        σ = 0.55′

The systematic error introduced in the simulation was +1.5′: the computation recovered it as +1.54′. And the standard deviation has dropped from 2.04′ to 0.55′, that is to the level of the random noise actually present in the observations. The three-unknown model has therefore explained the data far better. [C]

Left, the lines as observed; right, the same lines after removing the computed systematic error.

An honest warning. The three-unknown fix in this example falls 0.36 miles from the truth, against the 0.20 of the two-unknown fix: the better estimate of σ has not produced a better position. This is not a contradiction. When the azimuths are well spread, an error common to all the lines very nearly cancels in the computation of x and y, and the third unknown has little left to correct; what is gained is the diagnosis — knowing that the error is there and how large it is — and a standard deviation that tells the truth about the quality of the sights. With azimuths concentrated in one sector, on the other hand, the systematic error does not cancel at all, and then the third unknown really does move the fix. [C] [A]

The connection with the bisectors

One fact deserves to be known, because it ties the algebra to the graphical rule every navigator has been taught. With exactly three lines the three-unknown model has a single solution, and that solution is the point equidistant from the three lines, that is the meeting point of the bisectors of the triangle (the incentre, or one of the three excentres according to the senses of the azimuths). The numerical check is immediate and gives agreement to the sixth decimal place. [C]

The bisector method, always taught for the case of a suspected systematic error, is therefore not a graphical makeshift: it is the exact solution of the same problem that de Wit solves in general. With four or more lines the graphical rule no longer applies, whereas the computation carries on unchanged.

11.  Sights of unequal quality: weights

Not every line deserves the same confidence. A star low on the horizon suffers an uncertain refraction; a body taken at the edge of twilight, with an indistinct horizon, is worth less than one taken in good light; a sight taken from a rolling deck is worth less than one taken in a calm sea. Least-squares theory handles this case naturally, by giving each observation a weight w inversely proportional to the square of its expected uncertainty:

w(i)  =  1 / σ(i)²

The five sums become weighted sums, and everything else stays the same:

A = Σ w cos² Zn     B = Σ w sin Zn cos Zn     C = Σ w sin² Zn
D =
Σ w p’ cos Zn     E = Σ w p’ sin Zn

σ  =  √ [ Σ w r(i)²  /  ( n − 2 ) ]

How much it matters. In practice a coarse scale is enough — weight 1 for a normal sight, 0.5 for a doubtful one, 2 for a sure one — and the result changes little: weights act on the second decimal of a mile, whereas an error of one minute in an altitude acts on the first. They are worth using when one line is plainly worse than the others, not as an exercise in refinement. [A]

12.  When the method misleads

No computation makes bad observations good, and this must be said plainly, because the numerical form of the result — degrees, minutes and tenths — conveys a confidence that the raw data do not always justify.

  • A gross error in a single line is not rejected: it is spread over all of them. The method minimises the sum of the squares, so a line ten miles out pulls the fix towards itself. The only defence is the column of residuals: if one of them exceeds three times σ, that line is to be dropped and the computation repeated without it — but only if at least four lines remain, otherwise one is merely choosing the answer one likes.
  • With three lines random and systematic error cannot be told apart: any three lines form a triangle, and the triangle does not say where it came from.
  • With two lines the method reduces to the ordinary intersection, σ does not exist and G equals sin² of the azimuth difference: there is no check, and the number of decimals must not deceive.
  • Azimuths concentrated in a narrow sector give a fix that is very uncertain in the perpendicular direction, even with a small σ. G signals it before the damage shows.
  • The linearisation fails when the assumed position is far away: beyond thirty or forty miles the second pass is not a refinement, it is a necessity.

13.  The spreadsheet

The accompanying Excel workbook, « de_Wit.xlsx », contains the exercise of this article already loaded and ready to be altered. For each body only the data of the observation are entered — UTC, GHA, declination, corrected observed altitude, weight — together with the course and speed over ground and the assumed position; the sheet computes LHA, Hc, Zn, intercept, transport, the five sums, G, the fix, the residuals, σ, the error ellipse and the three-unknown model by itself.

The computation is repeated in three successive passes: the first starts from the assumed position entered, the second from the point obtained in the first, the third from the point obtained in the second. This is the iteration of section 8, made explicit in columns instead of hidden in a loop, so that the movement produced by each pass can be seen. The cells to be filled in are the only ones with a heavy border; all the others are formulae and are best left alone.

14.  In practice

The closing advice. Observe four or five bodies instead of three, choosing them by azimuth rather than by convenience — one per quadrant is worth more than five in the same sector. Compute the fix with the five sums, which take three minutes with any calculator and need neither paper nor tables. Then, before marking the position on the chart, look at two numbers: G, to know whether the geometry holds, and the column of residuals, to know whether there is a systematic error or a line to be thrown away. A fix accompanied by those two checks is worth far more than a point chosen by eye inside a polygon — and, unlike that one, it can be defended before anybody.

Sources

SourceWhat was taken from it
De Wit, C.: « Optimal Estimation of a Multi-Star Fix », Navigation — Journal of the Institute of Navigation, vol. 21, no. 4, Winter 1974-75, pages 320-325Authorship and conception of the method: least-squares estimation of the fix from several lines, in the plane tangent at the assumed position. [B]
De Wit, C.: « Optimal Position Estimate from a Star’s Fix with Correlated Errors », The Journal of Navigation, vol. 45, no. 1, January 1992, pages 126-133Treatment of correlated errors without prior separation of the systematic error; bias-free estimate of minimal spread. [B]
Kaplan, G. H.: « Determining the Position and Motion of a Vessel from Celestial Observations », U.S. Naval ObservatoryAttribution of the method to de Wit, independent development at the Royal Greenwich Observatory, adoption in Compact Data for Navigation and Astronomy and in the Nautical Almanac. [B]
Yallop, B. D. and Hohenkerk, C. Y.: « Compact Data for Navigation and Astronomy », Cambridge University Press / HMSOPublished form of the algorithm. [B]
The Nautical Almanac, commercial edition, pages 282-283The « many body fix » procedure with four iterations. [B]
The author’s own workComplete numerical example (ephemerides computed for 15 March 2026, verified in double precision), figures, error ellipse with the correct factor for an estimated σ, verification of the link between the three-unknown model and the bisector point, spreadsheet. [C] [A]

De Wit’s two original papers are available only on payment from their respective publishers; what is set out here is drawn from the publicly printed form of the algorithm and from least-squares theory, with independent numerical verification.

0 commenti

Invia un commento

errore: Content is protected !!