Thanks to sisve and jemaclus on #laravel (freenode) for helping me find the answer. I'm using TestCase::seed() to seed individual tables as part of the tests, which runs artisan, which loads all the commands, etc...

Artisan is also used by the DatabaseMigrations trait. And likely elsewhere. So it seems that wherever functionality is needed that is provided by artisan, $app->artisan is called up, which depends on Illuminate\Contracts\Console\Kernel being instantiated, which instantiates all registered Commands, etc...

Which involves a lot of classes and loading these chokes xdebug.

So the choices now seem to be to mitigate the problem as much as possible (avoid DI in constructors for one), or write replacement traits and override functions that decouple artisan (and it's loading process that is inefficient for Xdebug) from the actual functionality that's required.

Easier solution - a bit blunt but effective:

In app/Console/Kernel.php

/**
 * Remove all commands prior to bootstrapping when running unit tests
 */
public function bootstrap() {
    if (env('APP_ENV') == 'testing') {
        $this->commands = [];
    }
    parent::bootstrap();
}

That got it :-)

It may have worked, but seems more like a workaround than an actual "fix" to the problem. The constructors are called simply because the signatures are inside the classes. If the signatures were kept in the kernel where the class names are, it would not have to create instances to find which command you are calling.

Which, if you have a lot of DI, because you are testing your console commands, can quickly add up to a lot of startup bloat. Not to mention, the possibility that someone writes code in the constructor that "does something" - this will get run every time any command is run.