How do I modernize one part of an old system at a time?

Separate the system into functions and start with the part causing the most pain or risk. Add a new interface, service, integration, or component around that function while keeping the stable core in place. Test each step before moving to the next.