I guess sticking to the cave was the better option, then! Managed to keep everyone alive except for 3 people who are scripted to die.
I think only 3 people died in my game, as well. The daughter on the beach, and 2 people (a man and a woman who sat together on the ship) in the cave. The other survivors from the cave arrived in town on day 2 and said they had an easy time getting past the guard thanks to us.
Did 2 people die in the cave when you stayed there?
One of the men was dead on the beach on arrival and I could loot him. The old man's daughter is also dead on the beach and you can later do a quest to bury her.
Nobody dies in the cave, you get attacked by hungry lurkers but even if you don't do anything the survivors manage to gang up on them and defeat them with no issue.
The next day, one of your pals says that the woman stayed with her boyfriend and he wants you to help him rescue them. When you go there she is under attack but you manage to save her, the guy is dead though and you have to convince her to leave him behind.