I agree. The default referent of “only” is the number. So “yesterday I only caught five fish” with no extra stress is an expression of disappointment at not having caught more. 

Sent from my iPhone

On 28 Jan 2019, at 11:50, [log in to unmask] wrote:

> On Sun, Jan 27, 2019 at 08:09:14PM -0500, C. Brickner wrote:
> ----- Original Message -----
>> From: Stewart Fraser <[log in to unmask]>
>> To: [log in to unmask]
>> Sent: Sun, 27 Jan 2019 18:23:56 -0500 (EST)
>> Subject: Where is the opposite of "only" ?
>> “Yesterday I only caught five fish”
> One of my, admittedly trivial, pet peeves, the misplacing of "only". If someone says to me, "I only caught five fish,"  I could say, "So, you didn't take them home. You didn't cook them. You didn't eat them. You just caught them."
> The nit-picky statement should be "Yesterday I caught only five fish."

For what it's worth, to me, to get the meaning you mention here out of "Yesterday I only caught five fish" (i.e. that all you did was catch them; you didn't cook them or eat them, etc.), you have to stress the verb "caught". The closest you can come to a "neutral" interpretation (which still involves some amount of stress on the relevant constituent) is the interpretation where you expected to catch something more than five fish. 
- kechpaja -