Sam Hodgson gave the side a lead in the 18th minute before Hartlepool hit back with an equaliser just before half-time through Joe Grey.
Sam Hodgson then scored his second minutes into the second half to restore the lead with a penalty.
Ben Errington extended the lead to 3-1 with eight minute left before Jack Leonard made it 3-2, but it wasn't enough for the visitors at Mariners Park, with 445 fans in attendance.
The young Mariners will travel to Fulham in round three.
Keep up-to-date with our exclusive email newsletters.
Subscribe