The noise I've been experiencing is partially or fully coming from my power supply. I made a wall separating the transformer and OPS power supply board on one side and the buffer circuitry and all input, output and volume control wiring on the other side. The separator is a 2 mm aluminium sheet cut and bent to fit the components already fitted in place. The only wire that runs through this 'firewall' is the +/- 10 Volts supply from the OPS to the buffer board, through a tiny opening just big enough to let the three wires pass through. This brought down the self noise substantially. Later, I discovered that my transformer is vibrating a lot. It can be felt when placing the palm on top of the cabinet top cover. So I guess the next step is to use a better transformer. I usually use R Core but this time in my hurry to gather all parts I bought a ready made EI. It wasn't that cheaper than my usual custom ordered R Core, but it has so far turned out to be substandard. Two more lessons learned

Now I understand why transformers are potted in damping material.