Even if uncertainties exist, the outgroup can be chosen from organisms with many of the ancestral traits of members of the group to be studied--this will at least give a high chance of getting a useful outgroup. Further, the outgroup begins the analysis, which relies more on differences within the group after that, which helps make outgroup choice less critical for some of the analyses.
Work Step by Step
First think about the position of the outgroup--off the base of the group being studied. This suggests looking for ancestral traits widely shared in the group to be studied as a help in choosing the outgroup. Then, the outgroup is referred to less and less as one goes up the tree generated, making the outgroup choice less and less critical.