People do not stay with you forever. That is human nature. People are subject to change, whether in personality, interests or even beliefs.
Hence, do not get offended when people leave you because simply: you learn who is truly loyal to you. The decreasing amount of people may disappoint you but it should never affect your effort towards making your goals reality. It is your path, walk on it because only you will achieve the results. No one is going to make you successful but yourself. Whatever you do, make it ethical (always).