I guess it is totally normal behaviour as models are not purged after they are deleted from the database. You could propably refresh your relation also by calling:

$document->segments()->get()

but like I said this is pretty standard that relation models classes are not deleted when row in database is deleted.

I'm not sure but maybe this is your answer.
'segments()' is a function in your Document class. If you call it like $document->segments() you actually call that function and get a new QueryBuilder.
When calling 'segments' the object checks if the required data is allready laded. If yes it is returned, otherways it is queried from the database (like calling ->get() for the query).
$document->segments()->delete() will create a query which deletes the data in the database but it doesn't check the loaded data.
$document->segments->delete() should check the loaded data. If or not loaded it will be emptied and a query to the database will be send to clear it.

I could easily be wrong so please reply after you check it.