The solution I implemented for now looks like this in my user class:

public function projects(){
    $builder = $this->belongsToMany('App\Project','user_in_project_group');
    $builder->getQuery()->getQuery()->distinct = TRUE;
    return $builder->get();
}
        
public function projectGroups($project_id){
    return $this->belongsToMany('App\Group', 'user_in_project_group')
         ->wherePivot('project_id', $project_id)->get();
}
        
public function groupProjects($group_id){
    return $this->belongsToMany('App\Project', 'user_in_project_group')
        ->wherePivot('group_id', $group_id)->get();
}

This way I get always the values I need. On the downside, I always have to use brackets to access the values in my views. On top of that, my functions return collections, so if I want to use stuff like sync(), attach() or detach(), I need to implement additional functions which return a Relation object.