A simple implementation of what you are trying to do.

public function postVote($id)
{
    if( ! Request::ajax()) { 
        // Send humans home. Only AJAX zone !
        return Redirect::home(); 
    }

    $question = Question::find($id);

    if (is_null($question)) { 
        // Check for error code in AJAX response. It's faster !
        return App::abort(404); 
    }

    $voted = $question->votes()->whereUserId(Auth::id())->first();

    if(!$voted) {
        // If not voted, "vote up"
        $user = $question->votes()->attach($user->id);
        $question->increment('vote_cache');
    } else {
        // If already voted, "vote down"
        $question->votes()->detach($voted->id);
        $question->decrement('vote_cache');
    }

    $question->save();

    return Response::make($question->vote_cache);
}

So I need one or two (for Question and Answer) votes table ? Or does the votes field is an array ?

$voted = $question->votes()->whereUserId(Auth::id())->first();

In my current code, I don't have any votes table. I have only one "votes" integer field.

Votes() is the relation which you need to define on the pivot table "question_user". This will be a many - many relation as one user can vote many questions and one question can be voted by many users.

Table schema : id, user_id, question_id and of-course the foreign key relations...

Once done you can define your relation votes :

/**
 * Query the question's votes.
 *
 * @return Illuminate\Database\Eloquent\Relations\BelongsToMany
 */
public function votes()
{
	return $this->belongsToMany('User');
}

You can modify this a bit and make it work for answers also in the same table but that will increase the complexity. Keep two separate tables. Keep it simple !