Marcus: You are doing yourself violence, violence, my soul; and you will have no second occasion to do yourself honour. Brief is the life of each of us, and this of yours is nearly ended, and yet you do not reverence yourself, but commit your well-being to the charge of other men's souls.
The term 'violence' here is explained later in Book II:
Marcus: The soul of a man does violence to itself, first and foremost when it becomes so far as in it lies, a separate growth, a blain as it were upon the Universe. For to turn against anything that comes to pass is a separation from Nature, by which the natures of each of the rest are severally comprehended. Secondly, when it turns away from any human being or is swept counter to him, meaning to injure him, as is the case with the natures of those who are enraged. It violates itself, thirdly, when it is the victim of pleasure or pain; fourthly, when it acts a part, and says or does anything both feignedly and falsely. Fifthly, when, failing to direct any act or impulse of its own upon a mark, it behaves in any matter without a plan or conscious purpose, whereas even the smallest act ought to have a reference to the end. Now the end of reasonable creatures is this: to obey the rule and ordinance of the most venerable of all cities and governments.
In # 6, Marcus is scolding himself for doing violence to his own soul by straying from the Stoics' path. He explains this view of violence to the soul in #16 which I have reworked below. For clarity of meaning, I sometimes reworded a statement in the affirmative and sometimes kept the negative.:
Firstly: Take what comes, as it comes. In this way you will be one with Nature, able to understand the Universe, rather than to be separate from, and a detriment to, it.
Secondly: Violence to the soul also entails turning away from other human beings, or trying to hurt them, as do those who are enraged.
Thirdly: Don't allow yourself to become a victim, either of pleasure or of pain.
Fourthly: Don't be false or act a part.
Fifthly: Don't do anything thoughtlessly. Pay attention and be sure your actions are in accord with your purpose.
So in #6, Marcus is saying that he's been doing violence to his soul. He reminds himself again of the brevity of life, that there's little time left for him to do what's right, and admonishes himself for allowing others to have control over him.