Leave it alone. When something happens that dictates pulling the engine then do so. Messing with it now will trigger the notion that more and more is needed whilst you have the engine out and on an engine stand. Here are some consideration that may pop up:
a. As you say all gaskets and clean out plugs will invite replacement including timing gears cover.
b. While in there you will want to resurface the transmission's clutch release (throw=-out) bearing and maybe resurface/replace the pressure plate, new clutch dixc, etc.
c. Resurface the exhaust and intake manifolds and install new gaskets there.
d. Pull the head and do a valve job. Leading to a complete overhaul/rebuild of the engine.
e. Replace the timing gears.
f. And on and on and more and more. The temptation to keep doing things to the engine will become compelling. You started out just wanting to replace the pan gasket, clean and paint the engine.

You get the picture?
Sludge in the pan doesn't hurt anything unless it plugs up the oil pump screen and/or dippers.sufficient to starve those parts. Generally it just sits there in the bottom of the pan not bothering anybody.

Just run and enjoy it until it starts knocking.

It'll save you money on beer for your buddies.

Good luck with whatever your decision happens to be. You're welcome to my sound, sage advice.
Best,
Charlie