Ultimately the Potential belongs to the Account and not a contact. What happens if you work a Potential with a Contact, notes stored at Potential level are reflected in the Contact and then that Contact is no longer anything to do with the Potential? It then gets more complex when you have multiple Contacts associated with a Potential, you place a Note at Potential level which Contact should it be associated with all of them or a selection of them? This is why Potential notes are reflected at Account Level and not contact level.