You cannot un-equip the gear if your labor points fall below 1 for the current and any of the scheduled jobs..
If you do construction and you un-equip some of your gear, the reward (construction points) is diminished. I think this is true for all jobs.
But: if you started traveling, then un-equip your horse or horseback riding items, the time to travel remains the same (you keep your faster speed). If you equip those items after starting the journey, you remain at the lower speed.