Would it be wrong to send a gift and call it a day?

Unless you receive a written invitation, you are under no obligation whatever, but if you choose to send a gift that would be fine. You do not need to attend, and are very probably not expected anyway -- the "off the cuff" verbal invitation was a bit awkward at best.