Using public key cryptography, X adds a digital signature σ to a message M,…

Concept — The two keys of a public-key pair are mathematical inverses — whatever one key transforms, only its partner can reverse — so the key that is applied decides who is able to undo that step. Applying one’s own private key produces a value that anybody can undo with the matching public key, and that is exactly what makes it evidence of origin, i.e. a digital signature. Applying the other party’s public key produces a value that only that party can undo with their own private key, and that is what makes it confidential, i.e. encryption.

Two rules follow from this. A signature is always created with the signer’s own private key and checked with the signer’s public key, never the other way round. And when two such operations are layered, they unwind last-applied-first, exactly like nested wrappers.

Application — Trace the journey of the message M and its signature σ from X to Y.

1. X signs M by applying X’s private key. The result σ is a value that anybody can reverse with X’s public key, and that nobody but X could have produced.

2. X then encrypts the pair (M, σ) by applying Y’s public key. This is the outer, last-applied wrapper, and only the holder of Y’s private key can strip it.

3. Y receives the packet and removes that outer wrapper first, applying Y’s private key, because nothing else reverses a transformation made with Y’s public key.

4. Y now holds M and σ again, and checks the signature by applying X’s public key; recovering a value that matches M confirms that σ was produced with X’s private key.

Cross-check — Read the two goals separately; each goal fixes both of its keys, and the table must agree with the sequence derived above.

Contrast — Why the neighbouring orderings break down:

• A receiving side that begins with X’s public key would only make sense if the outermost wrapper had been created with X’s private key; here the outermost wrapper was created with Y’s public key, so it cannot be opened that way.

• A sending side that applies Y’s private key would require X to hold Y’s secret key. X never has it, so that step cannot be carried out at all.

• A sending side that begins with X’s public key encrypts the message to X rather than signing as X; the result could then be opened only by X and would carry no proof of origin.

• The last-applied-first rule is the fastest check: the sending side ends with Y’s public key, so the receiving side must begin with Y’s private key.

The sending side therefore uses X’s private key followed by Y’s public key, and the receiving side uses Y’s private key followed by X’s public key.