Prepare a grill to medium-high heat. Brush the plums and nectarines with the olive oil. Grill them for approximately 3 minutes per side or until they are nice and soft. Remove from the grill.
Put 4-6 scoops of ice cream on a serving piece or in individual bowls. Arrange the grilled fruit and berries over the top. Serve immediately.