If you've spent any significant time engaged in libertarian activism, or even if you haven't, you've probably heard people say things like, "Well, I have a life, I don't have time to play activist", or "those people are losers without real jobs or responsibilities", etc. I can't count how many times I've heard non-activists make comments like these in an effort to dismiss activists and their efforts. Living in the SF Bay Area where most people are politically on the left, I most often hear such complaints voiced against left-wing activists (often by right-wingers who are quite statist and not necessarily contributing much of meaning to society themselves), but it sounds much the same when they are voiced by statists against libertarians.
If you are a freedom activist, those stuck in authoritarian mindsets, or who have a perceived financial or other vested interest in maintaining statist oppression, will naturally tend to resent your efforts. So when they attempt to diminish those efforts, or discourage you, this should come as no surprise, and there's no reason to let it affect you. Know them for what they are, and let their words be like the wind whistling through a porcupine's quills.
Whether you experience your activism as work or play, as a moral duty or a rewarding activity, or as some combination of these things, does not determine the value of what you do. So the value of that activism certainly isn't determined by the mere words of some resentful critic who doesn't grok the importance of the struggle for freedom.
The reality is that except to the extent we are under the compulsion of government or some other external aggressor(s), we each have exactly the same amount of time, 24 hours in each day. Maybe you have health limitations that limit how you use some of this time, but aside from that, it is up to you.
If someone chooses to spend their time working at a job that does not advance the cause of freedom, so that they can have more money to buy things for themselves and live a comfortable life, or to support a small circle of those who depend on them, that is a valid choice. But it's not a choice that makes them more noble, worthy, or rational than those who choose to dedicate time and effort to the struggle for freedom. Krishnamurti said that it is not a sign of good health to be well adjusted to a sick society, and by a similar token, having high social status or the esteem of others in such a society is not necessarily a sign that you are making your best contribution to the world.
Never forget that to the extent a person works in the formal economy and is taxed by government, they are supporting State aggression. This does not make them a bad person – it is your right to live as you choose, and to engage in voluntary work for compensation, even if you know government will steal a portion of those earnings. For most people, this theft of their resources for evil purposes is involuntary, and to blame them for it is to blame the victims.
But to the extent they do not rely on the State to support them with resources coercively taken from others, a poor person who generates little income that finds its way into the hands of government agencies (or a well-to-do one who manages to prevent their resources being stolen and avoids patterns of consumption that feed the State), far from being a proper object of disdain or contempt, is worthy of appreciation and respect.
All else being equal, the life they are living is more, not less, moral than that of someone with a "respectable" job, hobbies, etc., who is indirectly contributing to aggression, even if involuntarily. If a person whose life choices not only leave the State with fewer resources with which to engage in aggression, but leave them with more disposable time, to the extent they choose to devote that time to libertarian activism and making the world a better place for not just themselves but everyone, they are deserving of our highest respect and admiration.
This activism can of course take many forms, and overlap with many other kinds of effort, work, or life activities, whether they are classified as work, hobbies, or something else, such as raising children to understand and appreciate the value of liberty. The latter in fact may be one of the most important contributions you could possibly make.
Whatever form(s) your activism may take, and whether it is a primary focus of your life, or something you try to fit in at the edges, to the extent you devote yourself to expanding freedom not just for yourself but for others, and your endeavors make a difference in the world, know that you are engaged in the highest form of community service.
If being a freedom activist is your calling, or to the extent you choose to make it your calling, know that there is no higher calling in the world. Regardless of the grumbling and attacks from those of little understanding, you could not be spending your time on this planet and in this plain of existence in any more commendable manner.