If your project is refactoring code without adding new features, is that time well spent?
How do you tell when you're finished?
How do you tell if you were successful, or even if you utterly failed?
How do you decide what needs refactoring?
How do you know that what you are refactoring will make things easier in the future?
Are past successes indications of future performance? In other words, will your future projects have the same problems that your earlier ones did?
What happens if you choose the wrong piece of code to refactor?
Don't spend money refactoring code until you know that you will see a benefit from the refactor. You've got better things to do with your time - like inventing new products. Otherwise, you've got a high likelihood of refactoring dead code.
Technical and Architectural Debt are terms used by software engineers as an excuse to rewrite code they don't like. Don't fall for it.