Packages is the way to go, especially for modules like your administration panel. Since it will be in your vendor directory, you can treat it like a separate project with proper version control. Use a branching workflow like this - http://nvie.com/posts/a-successful-git-branching-model/ to maintain upgrades and merge them back upstream to your package for future use.

Yes, packages are certainly the way to go here. What I'm currently working on is a yeoman generator that will use prompts to allow the user to specify which packages they'd like to include in the initial composer.json before yeoman runs composer install. With that model in mind, you could pre-fill your packages and just check off the ones you'd like to include.