> the explanation you provided about convergence, policy functions, etc. went over my head, sorry.
I was agreeing with you when I talked about convergence. You said that adding a cost would solve the problem and I agree that it does. I think you were probably thinking of a literal cost like "one cent". If you choose a cost like that then switching an infinite number of times has an infinite cost. You could instead think of the cost as a fraction of your expectation, the cost is 1% of whatever you end up getting back. Now if you switch an infinite number of times you end up having a cost of zero. That might seem counterintuitive, but recall that 1/3 is .3333 repeating. So when you sum 1/3 + 1/3 + 1/3 you get .9999 repeating. Yet 1/3 + 1/3 + 1/3 is equal to one. Infinitely close to something else is basically being the thing you are infinitely close to. Even though we never get the reward we know that the fraction is becoming infinitely close to zero. People call it "converging" when we have an infinite sequence we can sum to a real value. We know before ever seeing the value that we'll be multiplying it by zero. So we can refactor the equation to be 0*ev(switch) and then take advantage of the identity of 0x=0 to declare the result to be 0. Thus, the calculation converges to zero.
I was agreeing with you when I talked about convergence. You said that adding a cost would solve the problem and I agree that it does. I think you were probably thinking of a literal cost like "one cent". If you choose a cost like that then switching an infinite number of times has an infinite cost. You could instead think of the cost as a fraction of your expectation, the cost is 1% of whatever you end up getting back. Now if you switch an infinite number of times you end up having a cost of zero. That might seem counterintuitive, but recall that 1/3 is .3333 repeating. So when you sum 1/3 + 1/3 + 1/3 you get .9999 repeating. Yet 1/3 + 1/3 + 1/3 is equal to one. Infinitely close to something else is basically being the thing you are infinitely close to. Even though we never get the reward we know that the fraction is becoming infinitely close to zero. People call it "converging" when we have an infinite sequence we can sum to a real value. We know before ever seeing the value that we'll be multiplying it by zero. So we can refactor the equation to be 0*ev(switch) and then take advantage of the identity of 0x=0 to declare the result to be 0. Thus, the calculation converges to zero.