"file is encrypted or is not a database"
There's a version conflict between the Trac database and the accessing sqlite version. The best solution I found while searching the web, was to convert the databases manually:
sudo -s
apt-get install sqlite
cd /var/your-trac-project/db
cp trac.db trac.db.orig
sqlite trac.db .dump > trac.sql
rm trac.db
sqlite3 trac.db ".read trac.sql"
chown www-data:www-data *
trac-admin .. upgrade
After that, Trac worked again.