By default, when a search times out, ANT will close the channel. So search timeout is what controls this behavior. If your search timeout is 10 seconds, then after 10 seconds of not finding anything, the channel will close. If your search timeout is infinite (set to 255) then the channel will never close automatically and search until it finds something.
It is important to remember that searching has very negative consequences on battery life, in particularly in coin cell applications, so infinite search is almost always a bad idea on battery operated devices.
At the application layer, you could specify to close the channel after a certain number of Rx fails, but this is not built into ANT.
You should also be aware that there are two searches - low and high priority, with the difference being that high priority can block other channels (if they exist) while low does not. You need to set both to known values. I also believe that low priority search runs first if both types are enabled. You can play with this in ANTware under the advanced tab.