You can use the Binary Classification Inspector to adjust the threshold and see how your confusion matrix (and associated metrics) change. Once you arrive at a reasonable threshold value, you can apply a threshold as you mention using the Rule Engine node.
Check out this discussion from @paolotamag and associated workflow: