Page 1 of 1

Database corruption

PostPosted: Tue Oct 15, 2024 2:57 pm
by kashinc
Hello,

I am getting database corruption randomly on different clusters, it seems that somehow characters that are not supported by the DB are getting put into the database. I am assuming that its openers in putting characters that are not supported by the database. after running a mysql repair command it always seems to fix the issue.

Has anyone else ran into this issue and has anyone found a solution to fix it or prevent it from happening?

Re: Database corruption

PostPosted: Tue Oct 15, 2024 6:03 pm
by carpenox
its more than likely being exploited through the latest RCE - what svn are you using?

Re: Database corruption

PostPosted: Tue Oct 15, 2024 10:07 pm
by kashinc
The SVN I am currently running on is 3875

Re: Database corruption

PostPosted: Wed Oct 16, 2024 9:55 am
by carpenox
ok so it wasnt a hacker, are you using a VPS or bare metal? ive seen this often on VPS, another reason the vicidial group strongly suggests NOT to use them.

Re: Database corruption

PostPosted: Wed Oct 16, 2024 11:42 am
by kashinc
all bare betal servers

Re: Database corruption

PostPosted: Wed Oct 16, 2024 12:50 pm
by carpenox
im not sure ive never came across it, try asking in the group chat we're in

Re: Database corruption

PostPosted: Wed Oct 16, 2024 10:39 pm
by williamconley
kashinc wrote:Hello,

I am getting database corruption randomly on different clusters, it seems that somehow characters that are not supported by the DB are getting put into the database. I am assuming that its openers in putting characters that are not supported by the database. after running a mysql repair command it always seems to fix the issue.

Has anyone else ran into this issue and has anyone found a solution to fix it or prevent it from happening?


"Characters that are not supported" in which field(s)? Does this happen during load of new leads? (If so: How are you loading the leads?)

What makes you think that unsupported characters are being inserted? The actual error message (from your next occurrence ...) may shed some light.

Are the tables which become corrupted ... Large? EG: Over 10Million records? Are they logs or other tables?

Re: Database corruption

PostPosted: Wed Oct 16, 2024 11:07 pm
by kashinc
williamconley wrote:
kashinc wrote:Hello,

I am getting database corruption randomly on different clusters, it seems that somehow characters that are not supported by the DB are getting put into the database. I am assuming that its openers in putting characters that are not supported by the database. after running a mysql repair command it always seems to fix the issue.

Has anyone else ran into this issue and has anyone found a solution to fix it or prevent it from happening?


"Characters that are not supported" in which field(s)? Does this happen during load of new leads? (If so: How are you loading the leads?)

What makes you think that unsupported characters are being inserted? The actual error message (from your next occurrence ...) may shed some light.

Are the tables that become corrupted ... Large? EG: Over 10 Million records? Are they logs or other tables?


It is almost as if the openers are modifying the leads and putting in characters that are not supported, for example, characters with four bytes or weird non supported characters

I found these characters in the custom fields when I noticed DB corruption. There is no error message; dialing stats starts going down until they show 0 dials in the carrier stats 1 min box. This is generally an indicator that DB corruption or a heavy lock-up has occurred.

The files being loaded are less than 10 million; they aren't even 1 million records. The lead has a character, and once it hits that lead, the system does not return to normal until I run a db repair command. The leads also need to be removed before doing this, or else once it hits that lead again, we are back to square one.

Hope that is enough information.

Re: Database corruption

PostPosted: Thu Oct 17, 2024 1:43 pm
by williamconley
Are your agents copy/pasting information from other web pages during the interview?

It would also be possible to run a query that cleans these fields on a timed basis, or at least locates these characters so you may begin to narrow down the source and actually troubleshoot once found.