Gotcha. What I am describing below is a new idea for me and I have not used it myself (yet)...I did test it in a sandbox report and I think the logic is sound but please test this idea more thouroughly for yourself.
Create a formula field as "Fail Percentage" (or whatever you want):
if {table.field}=failed then 100 else 0
//the red is your condition to know if a row is counted as failed
Now Insert a Summary on this formula field as an Average at the group level of IP Address.
This should give you the same number / percentage value as your other formula (CountofFailed/TotalCount) but allows you to use it in the TOP N Group Sort Expert.
Let me know if it works for you as this pesky issue seems to come up a lot
