How do I know if an automation is actually saving time instead of adding another tool?

Compare the new process with the old one using total staff time, errors, waiting time, and how often people must fix the automation. Include the time spent monitoring and maintaining it. Keep it only if the net result is better.