Solution: Let $q\in E\setminus \{p\}$. We need to show that there exists $\epsilon >0$ such that \[B(q,\epsilon )=\left\{ x\in E:d(q,x)\lt\epsilon \right\} \subset E\setminus \{p\}.\]

Since $p\notin E$, the distance between $p$ and $q$ is not zero, so we have $d(p,q)>0$. As $E$ is open and $q\in E$ we can find $r>0$ such that $B(q,r)\subset E$. Define \[ \epsilon = \min \left\{ \frac{d(p,q)}{2},r \right\}>0. \]

$r\lt d(p,q)$

$r\gt d(p,q)$

Then, we claim that $B(q,\epsilon )\subset E\setminus \{p\}$. We only need to show that $p\notin B(q,\epsilon )$. If it is, then \[ d(p,q)\lt\epsilon \leq \frac{d(p,q)}{2}, \] a contradiction. Thus, we have $B(q,\epsilon )\subset E\setminus \{p\}$ and hence $E\setminus \{p\}$ is open.

Note: If $E$ is closed, then $E\setminus\{p\}$ is not closed. For example, take $X=\mathbb{R}$ with the usual Euclidean metric and $E=[0,1]$. Then $E\setminus \{0.5\}=[0,0.5) \cup (0.5,1]$, which is neither closed nor open.