Consider two rotations r and r' in three dimensional space R3 (around two axis passing through the origin). It is a known (but not trivial) fact that the composition r'or is another rotation. The question arises naturally how to determine its axis and angle. Is there an easy geometric way to do it?
Until recently I thought that the answer is "no", because I had never heard of it in any lecture on geometry or linear algebra, neither read it in any textbook or on the internet (see for example here). So it was a very pleasant surprise to find a simple geometric construction in Penrose's book The Road to Reality! He says that it was actually Hamilton's idea (but it seems to me that it has been forgotten by generations of teachers). Here it is. And it doesn't need higher maths; any interested high school pupil can understand it.

Main idea: Rotations "add" like vectors

Consider the unit sphere, i.e. the sphere with radius 1 and center O. A "great circle" is the circle that you get when you intersect the unit sphere with a plane containing O. Let us call "great arc" any oriented arc of a great circle. In the following figure I show three great arcs, namely ⟨AB⟩, ⟨BC⟩ und ⟨AC⟩. They form the so-called "spherical" triangle ABC.

Composed rotations in euclidian three space

The arc-length of ⟨AB⟩ is equal to the angle AOB (measured in radian).
Let us denote by rAB the rotation whose axis is perpendicular to the plane containing the great arc ⟨AB⟩, whose angle is the twice the arc-length of ⟨AB⟩ and whose direction is according to the orientation of the arc. Then we have the following astonishing theorem.

Theorem: rBC o rAB = rAC

This means that the composition of rotations is similar to vector addition! It suffices to replace vectors by great arcs representing the rotations (always with half angle).

The proof of this theorem is not difficult. Consider the points B' and C' that are symmetric to B and C with respect to A; the points A'' and C'' that are symmetric to A and C with respect to B; and finally the points A''' and B''' that are symmetric to A and B with respect to C.

What happens if you make a rotation after another?

Thus we get three congruent copies of the spherical triangle ABC, namely AB'C', A''BC'' and A'''B'''C. The rotation rAB sends the first to the second, and the rotation rBC the second to the third; therefore the composition rBC o rAB sends the first to the third. But the rotation rAC does the same, and since the three vertices of a spherical triangle suffice to determine the "position of the sphere", it follows that rBC o rAB = rAC !

Consequences

Here are some corollaries, that follow from the theorem (but are not mentioned in Penrose's book).

  1. The composition of rotations is a rotation.
  2. Composing rotations is non-commutative, i.e. in general the order matters. For exemple with the above notations we have
    rBC o rAB = rAC  but  rAB o rBC = rBA'' o rC''B = rC''A''
    More precisely we can say that inverting the order changes the axis but not the angle! In fact rAC and rC''A'' have different axis but share the same angle. (One can prove this by matrix calculation too, using the fact that the trace of a product does not depend on the order in which you write the factors.)
  3. Triangle inequality: The angle of the composition is smaller or equal the sum of the angles of both rotations. And there is equality if and only if their axis and direction coincide. For example if you compose a rotation of 20° with one of 30° then you will never get one of angle 51°.

The third corollary seems interesting to me. I have never seen it in any lecture and I don't have any idea how one could prove it with linear algebra tools (matrix calculation).

By the way, in hindsight the property that one has to take as rotation angle the double of the arc-length isn't a surprise to the connoisseur! It reminds of the fact that you need 720° (and not 360°) to perform a "topologically zero rotation". For this read what I wrote here about the fundamental group of SO(3) or, what amounts to the same, three dimensional projective soace.

Another method: Using reflections

Here is another way, which you may already know, to compose rotations geometrically. It is beautiful too, but not is not as powerful as the method described above. Let us denote by sF the reflection with respect to the plane F.

Composition of reflections in 3 space

The above drawing shows that the composition  sH o sF sends the basis (i,j,k) to (i',j',k') by a rotation around the axis i and with twice the angle bewteen the planes F and H. So we have:

Lemma 1: The composition of two reflections is a rotation. The rotation axis is the intersection of both planes; the angle is twice the angle between the planes.

Note that  sH o sF   is the inverse rotation of  sF o sH  ; both have same axis and same angle, but opposite directions.

Lemma 2: Every rotation can be written as the composition  sFo sH  of two reflections.

For the proof of lemma 2 choose an arbitrary plane H through the rotation axis. Then there exist exactly two planes through the rotation axis that form with H half of the rotation angle. One chooses as second reflection plane F the one that gives the desired direction of rotation.

There is a cruial point: The choice of the first reflection plane is free (as long as it contains the axis). And of course one can also begin with the choice of the plane of the second reflection.

Now let r and r' be two rotations. Let H be the plane containing both axis. As we have seen above there exist planes F and F'such that

r = sH o sF  and  r' = sF' o sH

Hence

r' o r   =   sF' o sH o sH o sF   =   sF' o sF

Here we used that fact that the two reflections sH cancel each other. What remains is the composition of two reflections and therefore a rotation.

This shows that the composition of rotations is a rotation. And it also allows to find the axis and angle, but drawing it on a paper is not as clear as with the first method.