I held my zebra finch babies again?

ok so my zebra finches have babies they have their feathers but still cant fly and are still in the nest being fed by their parents when i changed the parents food the cage door closed and scared the babies out of the nest they jumped out and the parents started panicking i put them back in but i had to hold them to put them back in this happened already 2 times and if the parents get upset will they abandon them it happened twice and im worried