As I promised, I rewrote parts of Python QAM modem to use complex numbers, which leaves code much shorter and clearer (provided that you understand complex numbers).
Also, I put the common code in a separate file, to avoid redundancy in TX and RX parts. The FIR lowpass filter was improved too.
The files are m2co.py, m2tx.py and m2rx.py. All of them can be found at https://epxx.co/artigos/modulation.
I think I went far enough with this. I was planning to write a "version 3", but I am not sure I will. Versions 1 and 2 already make good classroom examples. FWIW I will tell what's in my mind for v3, maybe someone else picks it up.
First, the "classical" demodulation approach showed its limits in version 2; baud rate can not go above 1000 or 1100 Hz for a carrier of 1800 Hz. The FIR filter has a trade-off between frequency precision and time-domain precision.
Increasing baud rate demands increasing frequency precision, but then symbols begin to smear together after filtering. In order to reach V.29 or V.32 baud rate (which is almost the same as carrier frequency), some other detection approach must be employed.
I played with some formulas, remembered some trigonometry identities at Wikipedia, and came up with one idea.
Since the derivative of QAM signal is well-defined by a simple formula, and there are two unknown variables that shape the QAM signal — amplitude and phase — it is possible to "invert" the formula and find amplitude and phase given two derivatives taken from time-domain signal.
The signal derivative is at hand; it is just the difference between two consecutive samples. If we measure two derivatives 90 degrees apart from each other, formulas can be further simplified.
This approach was tested on spreadsheet, and should work in a Python implementation. It seems possible to determine symbol in half carrier wavelength at the most, normally a quarter wavelength is enough, which means that decoding is possible when baud rate approaches or surpasses carrier frequency.
Another advantage of this technique is more resilience to small carrier deviations. They would appear just as a very small but persistent phase delay in every sample, which can be ignored or compensated for.
I have no idea if this is the technique that advanced modems actually use, but I bet it is at least some variation of this basic approach.