All in all I'm doing ok, although the v18.x release is taking longer than I'd like as I have a huge backlog of stuff I really want to get onto! Regardless, it sounds like I'm doing better than you! :/
Whist the message "File ./ib_logfile0 was not found" seems relevant, IMO it shouldn't actually be an issue. AFAIK the ib_logfileX files are where MariaDB stores historical DB changes so that if something bad happens, your DB can be restored - including DB changes occurring immediately prior to the crash - which may not have yet been written to the DB. I'm not sure, but the fact that it's looking for them, suggests to me that it may have previously crashed or not been shutdown cleanly prior to this issue?
It's likely cold comfort for you, but FYI I just launched a fresh v18.0 Redmine and doesn't have any ib_logfileX files and MariaDB is starting fine:
Apr 30 23:06:39 JED-TEST-REDMINE-180 systemd[1]: Starting mariadb.service - MariaDB 10.11.6 database server...
Apr 30 23:06:41 JED-TEST-REDMINE-180 mariadbd[397]: 2024-04-30 23:06:41 0 [Note] Starting MariaDB 10.11.6-MariaDB-0+deb12u1 source revision as process 397
Apr 30 23:06:41 JED-TEST-REDMINE-180 mariadbd[397]: 2024-04-30 23:06:41 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
Apr 30 23:06:41 JED-TEST-REDMINE-180 mariadbd[397]: 2024-04-30 23:06:41 0 [Note] InnoDB: Number of transaction pools: 1
Apr 30 23:06:41 JED-TEST-REDMINE-180 mariadbd[397]: 2024-04-30 23:06:41 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Apr 30 23:06:41 JED-TEST-REDMINE-180 mariadbd[397]: 2024-04-30 23:06:41 0 [Note] InnoDB: Using liburing
Apr 30 23:06:41 JED-TEST-REDMINE-180 mariadbd[397]: 2024-04-30 23:06:41 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
Apr 30 23:06:41 JED-TEST-REDMINE-180 mariadbd[397]: 2024-04-30 23:06:41 0 [Note] InnoDB: Completed initialization of buffer pool
Apr 30 23:06:41 JED-TEST-REDMINE-180 mariadbd[397]: 2024-04-30 23:06:41 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
Apr 30 23:06:41 JED-TEST-REDMINE-180 mariadbd[397]: 2024-04-30 23:06:41 0 [Note] InnoDB: End of log at LSN=1841905
Apr 30 23:06:41 JED-TEST-REDMINE-180 mariadbd[397]: 2024-04-30 23:06:41 0 [Note] InnoDB: 128 rollback segments are active.
Apr 30 23:06:41 JED-TEST-REDMINE-180 mariadbd[397]: 2024-04-30 23:06:41 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
Apr 30 23:06:41 JED-TEST-REDMINE-180 mariadbd[397]: 2024-04-30 23:06:41 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
Apr 30 23:06:41 JED-TEST-REDMINE-180 mariadbd[397]: 2024-04-30 23:06:41 0 [Note] InnoDB: log sequence number 1841905; transaction id 3242
Apr 30 23:06:41 JED-TEST-REDMINE-180 mariadbd[397]: 2024-04-30 23:06:41 0 [Note] Plugin 'FEEDBACK' is disabled.
Apr 30 23:06:41 JED-TEST-REDMINE-180 mariadbd[397]: 2024-04-30 23:06:41 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Apr 30 23:06:41 JED-TEST-REDMINE-180 mariadbd[397]: 2024-04-30 23:06:41 0 [Warning] 'innodb-file-format' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
Apr 30 23:06:41 JED-TEST-REDMINE-180 mariadbd[397]: 2024-04-30 23:06:41 0 [Warning] 'innodb-large-prefix' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
Apr 30 23:06:41 JED-TEST-REDMINE-180 mariadbd[397]: 2024-04-30 23:06:41 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
Apr 30 23:06:41 JED-TEST-REDMINE-180 mariadbd[397]: 2024-04-30 23:06:41 0 [Note] Server socket created on IP: '127.0.0.1'.
Apr 30 23:06:41 JED-TEST-REDMINE-180 mariadbd[397]: 2024-04-30 23:06:41 0 [Note] /usr/sbin/mariadbd: ready for connections.
Apr 30 23:06:41 JED-TEST-REDMINE-180 mariadbd[397]: Version: '10.11.6-MariaDB-0+deb12u1' socket: '/run/mysqld/mysqld.sock' port: 3306 Debian 12
Apr 30 23:06:41 JED-TEST-REDMINE-180 mariadbd[397]: 2024-04-30 23:06:41 0 [Note] InnoDB: Buffer pool(s) load completed at 240430 23:06:41
Apr 30 23:06:42 JED-TEST-REDMINE-180 systemd[1]: Started mariadb.service - MariaDB 10.11.6 database server.
I was able to recreate the issue you hit, by starting MariaDB manually and then killing it (i.e. not cleanly), reinforcing my suspicion that your DB did not close cleanly prior to this issue.
I have a (perhaps dirty?) fix/workaround. It may not include recent changes but I'm not sure on what else there is to do. It assumes that the only DB you want to keep is 'redmine_production' (which should be the only one required - unless you've created some custom DBs). If you want others, please rerun the "mysqldump" line with "redmine_production replaced with the name of the desired DB name.
# manually start mariadb ignoring errors
mariadbd-safe --innodb-force-recovery=6 &
# dump the redmine production DB
mariadb-dump redmine_production > redmine_production.sql
# kill the emergency mariadb process
kill $(pgrep mariadb)
# double check it's not running
pgrep mariadb
# if that gives NO result then you're good to continue. If it outputs a number, try the pgrep command until it does
# move the broken DB out of the way
mv /var/lib/mysql /var/lib/mysql.bak
# reinstall mariadb
apt install --reinstall mariadb-server
# it should now start
systemctl start mariadb
# double check it's running
systemctl status mariadb
# recreate redmine DB and load from dump
mariadb-admin create redmine_production
mariadb redmine_production < redmine_production.sql
# get current redmine DB password & recreate DB user
DB_PASS=$(grep -A 10 'production:' /var/www/redmine/config/database.yml | sed -En "\|password:|s|^.*: (.*)|\1|p")
mariadb --execute "grant all privileges on redmine_production.* to redmine@localhost identified by '$DB_PASS'; flush privileges;"
# optionally remove the old DB data dir - don't do that if you want to try to properly recover it later (although I'm not sure how!?)
rm -r /var/lib/mysql.bak
Hopefully Redmine should be working again!
If not, then TBH, I'm not sure. Perhaps if you answer these questions I might think of something?:
Is this a "clean" v18.0, or have you restored your old data (and/or database) from your old instance?
If it's from data from an older TKL instance, what TKL version did it come from?
If from an older TKL instance was it a TKLBAM restore or a manual one?
If TKLBAM, was it a complete restore or a partial one?
Regardless If a manual restore or partial TKLBAM, what files/directories did you restore?
Regardless, has this server crashed and/or was MariaDB not shutdown cleanly?
Hi there! :)
All in all I'm doing ok, although the v18.x release is taking longer than I'd like as I have a huge backlog of stuff I really want to get onto! Regardless, it sounds like I'm doing better than you! :/
Whist the message "File ./ib_logfile0 was not found" seems relevant, IMO it shouldn't actually be an issue. AFAIK the ib_logfileX files are where MariaDB stores historical DB changes so that if something bad happens, your DB can be restored - including DB changes occurring immediately prior to the crash - which may not have yet been written to the DB. I'm not sure, but the fact that it's looking for them, suggests to me that it may have previously crashed or not been shutdown cleanly prior to this issue?
It's likely cold comfort for you, but FYI I just launched a fresh v18.0 Redmine and doesn't have any ib_logfileX files and MariaDB is starting fine:
I was able to recreate the issue you hit, by starting MariaDB manually and then killing it (i.e. not cleanly), reinforcing my suspicion that your DB did not close cleanly prior to this issue.
I have a (perhaps dirty?) fix/workaround. It may not include recent changes but I'm not sure on what else there is to do. It assumes that the only DB you want to keep is 'redmine_production' (which should be the only one required - unless you've created some custom DBs). If you want others, please rerun the "mysqldump" line with "redmine_production replaced with the name of the desired DB name.
Hopefully Redmine should be working again!
If not, then TBH, I'm not sure. Perhaps if you answer these questions I might think of something?: