I think the easy way would be setting the dynamic container to have a single document. By that way if he tries to open second case, the first one would be removed from the thread list. OOTB.
But this way the user will not be able to open two different case types at once as well.
Other way would be, call one java script from the link of the assignment handle, then make safe url call to an activity. In that activity check if the user already opened any case or not.(You can put all your custom logic in the activity). Now based on the activity status, you can either open the assignment or throw an alert.
Please check out for PRRequestor API methods to get the thread list for a single requestor.
In the portal main content area, we need to provide dynamic container to hold document(Previously known as work area). It can be of two mode: single doc or multiple doc. You should select single document.