DataBase Server Specs (Advise really appreciated)

[gequiros]

I need worst case scenario for this Database server, i know a lot of factors come in to place... but, how many agents can i move with this toy:

Motherboard: 1x X9DRi-LN4F+
- Onboard NIC: Integrated Dual Intel 1000BASE-T Ports
- Management: Integrated IPMI 2.0 Management
- PCIe: 1x Full Height PCI-e x8 slot
CPU: 2x Intel Xeon E5-2660 V1 Octa (8) Core 2.2Ghz
Memory: 128GB DDR3 (8 x 16GB DDR3 -REG 12800R HP )
Drive: 512GB SSD

What do i need to improve ? and how many agents as it is can handle with a dialing ratio of 15 ( Let's assume worlds crappiest leads )

I will use 20 Dialing Servers and 3 Web Servers ( 1 to do load balancing and 2 real web servers )

I need advise before i purchase and take decisions...

[williamconley]

I'd love to give you a number. But the reality of the situation is that every room is different and hardware specs ALL matter (front side bus, HD speed, RAID level and hardware speed and number of drives, 3rd party apps loading leads or viewing "wall" data on a screen for the agents, replication ... literally everything matters).

So while you may get an answer from someone, they'll be guessing (educated or otherwise) and they'll be wrong. You'll find that out when you load it up and the number differs. Keep an eye on it as you load it up, and see if you can find out what could be offloaded to another server or altered to reduce load (indexing, replication for reports ...). Eventually you'll find the absolute top, but you may never get there. You may also be paying enough attention to see the levels rising and take action to split your cluster.

Then you'll tell people who purchase similar hardware how many agents you managed before overload and wonder why theirs is so different from yours. Because everything matters.

That being said: Even at 15:1 dialing you ought to be able to exceed 150 agents. Perhaps 200. But don't quote me. I'm wrong, too.

[gequiros]

What Config would you suggest me to reach 400 agents ? and how many dialers as well Web servers ?

Thanks, and sorry for hijacking your time... as always, really appreciate the help of any of you guys...

[williamconley]

Honestly: If you're planning to put down a system with 400 agents on it, you need to Contact The Vicidial Group. They are the hands-down most qualified to make sure your system works at launch and beyond.

I'd personally consider it irresponsible to not at least have a Professional Vicidial Consultant directly involved in the system buildout (if not actually being the ones building it out for you ...) for a system that size with that many paychecks and families riding on it.

Remember what I said: Your experience WILL differ from any advice you get here based on a wide array of criteria. If you have the ability to build it out slowly as you grow, you'll have a much better experience. I hope you have a whole lot of bandwidth, too. Seriously.

That being said: You'll want to start with Four Powerful Dedicated webs (100 agents per web) 1 Seriously Powerful DB (with a six drive RAID10 on 15kSAS drives with 6G RAID and 6G RAID cables) backed by a dedicated replication server for reporting. 1 Archive server to offload all recordings and backups. Good archiving and pruning rules for all log tables and recordings. And at least four (also Seriously Powerful) Dedicated Dialers or six (Not Quite as Powerful as the dB) Dedicated dialers. You can then load one of each of your server types until they overload and use that experience to determine if you have enough before adding on.

Be sure each Dialer has a dedicated public IP with a minimum of 10M up & down bandwidth (preferably 20). Be sure all dialers have 1G direct access to each other (all on the same local network on the same 1G switch for internal communciations). Be sure you configure their Vicidial IPs with the Internal IP address. Agents can access via public or private IPs, but they must communicate with each other privately on 1G speed network preferably without any iptables or other firewall interference (I don't even like managed switches in this network, as that's just one more link to fail!).

But seriously: Get someone who knows Vicidial involved in your buildout. It's been known to cost jobs. Like: IT jobs. Possibly YOURS.

[gequiros]

I hear ya !!!

you are the expert... i am just planning it on the go

What's your definition for powerful DataBase Server in terms of CPU and Memory ( I guess DDR4 ), i've been reading everywhere, some suggest dual XEON some suggest SINGLE processor, i am getting clueless..

As well, what's your definition of Powerful dedicated Dialers, i heard they need around 8gb DDR

At this point, it is all about planning and doing the major figures on hardware and cost, if not, i will lease servers on a data center, and postone this for 6 months while i grow and increase cash flow...

Thanks so much... i mean it...

[williamconley]

More cores = more power. The number of CPUs used to get more cores is relevant only insofar as more CPUs tends to be more cores. (1 CPU with 4 cores is the same as 2 CPUs with 2 cores ... it's the core count that matters)

Faster cores = more power.

More memory is more power.

Faster memory is more power.

Faster front side bus is more power.

Faster (NOT bigger) hard drives is more power. This speed is not just a factor of the HD speed itself, the drive controller and front side bus are also involved. And the drive controller speed can be limited by the drive cable. 6G RAID 6G Cable and 6G HD all are needed to achieve 6G. And then HD itself should be able to sustain fast throughput for lenghty periods of time (thus 15K SAS is a reliable source ... some SSDs are also capable).

Total HD space should be over 100G (preferably 300G) to be sure that in the long term your DB and logs never exceed 75% of the drive space. Nothing else actually needs to reside on the DB server, but the dialers will need at least temporary storage for each day's recordings (if you're recording calls, that is).

If you can get a 16 or 20 core (not "virtual" cores, but physical) server at 3.0 Ghz with 32G of RAM and six 600G 15k SAS drives in RAID 10, you'll be doing good. You can start much lower and switch out the DB server as you grow, of course. Most start with 8 cores and less memory and add/change as their system grows. Monster servers are expensive.

Do not get the "bleeding edge" best newest servers. Linux drivers are often not available for them. Be sure they have linux drivers available (which is only possible by performing an install in many cases ... so buy one and install the OS instead of buying 10 at your first hit).

[gequiros]

Abusing of you a bit, lol.. i've look around for:

Database:
Processor - 2 x AMD Opteron 6386 SE 16-Core 2.8GHz Processors ( Would be 32 cores )
Memory - 192GB Ram DDR3

Or:

2 x E5-2690V4 2.6GHZ ( 28 Cores )
Memory - 128GB DDR4-2133

i guess by that i "would be safe"... Which one would be your pick ?

Intel XEON or AMD Opteron ? ( I guess you would call for XEON )

Or do you suggest me a better one, i am open to suggestions

Which Raid Card should i go, as well, Hard Drives...
i thought you would suggest me going for: M.2 NVMe
I've been thinking going for 2 x 512gb Samsung 960 PRO or going for 2 x 480gb Intel Optane 900P in raid 10 ( would that be better than SAS or it would be a waste ? )

On my actual DB server the DataCenter put a KINGSTON SV300S3 ( max read / write speed 500 /260 MB/s ), that's regular SSD, and i don't think right now that's causing any hog or bottlenecks... i guess, lol...

For dialers and voice recording, would normal 7200 rpm drives work fine ( this would be temporary), or should i go with at least 10K or 15k would be a must ?

What would be the ideal CPU for a Dialer server ?

I guess file server does not need to be powerful, any 8 core with decent 8gb memory would do the trick and all it matters is storage reliability, correct ?

i am expecting to crank it up in 2 - 3 months this OP... need to build cash flow...

Thanks my man for your help !!!

[williamconley]

You're on your own there. We only decipher to that level for clients. There's no way to know which will have superior performance beyond the basic specs. Drivers are for the motherboard (chipset), and not for the CPU itself. Compatibility (safety) is about the chipset, but performance is about the whole enchilada. Get the best price break for the Speed and core count (and all other speed factors) that you can. But remember that while there's no gurantee Vicibox will install smoothly, older systems have better odds at having OpenSUSE drivers available.

Matt has specified the RAID on this forum many times. There are a lot of RAID systems out there, and he doesn't recommend any of them except the one that he knows works because that's the backbone of the DB and many RAID fail under consistent load (there have been a lot of posts for people with RAID failing to maintain).

I'm not going to quote Matt's recommendation, you can search the forum for it in Matt's own words. When we use RAID it's built in (stock) PERC RAID in our Dell servers (we ONLY use Dell for our Rack servers, and don't support RAID in our starter-level non rack Standard servers). The PERC raid servers behave marvelously for us, and we've no reason to look further.

We've had several clients send us other servers with other RAID controllers in them, but those "NonPoundTeam hardware" servers and clusters are usually not heavily loaded, thus not a good comparison. We have configured them for the clients in question (even when the configuration is ... arcane, lol), but we don't generally recommend them until we know they will handle "full bore every day all day", which is just not something we can say we've observed with client servers we did not provide the hardware for. For many of those, we're merely the Host (no babysitting )

[mflorell]

Just a note about Dell PERC RAID cards, they are NOT built for high load Linux use. We have had to replace several high-volume system Database RAID cards for clients with LSI Logic MegaRAID caching RAID controllers in order to fix overheating or throughput limitations that happen when Dell PERC RAID cards are run past their designed specifications.

[gequiros]

Thanks so much for such a great advises...

Will we gain too much performance using: M.2 NVMe 4 x 480gb Intel Optane 900P in raid 10 or regular SSD by Samsung will do the trick, or i should remain loyal to SAS 15k ?

[frequency]

Thanks so much for such a great advises...

Will we gain too much performance using: M.2 NVMe 4 x 480gb Intel Optane 900P in raid 10 or regular SSD by Samsung will do the trick, or i should remain loyal to SAS 15k ?

I just moved a client from E5-1650v3, 240x2 SSD, HWR 1, LSI 9260i card to a AMD EPYC 24c/48t server with 128G DDR4 ram with 2x960 M.2 NVMe drives (no HW raid). the new MySQL DB was fetching the same stats in 15s with lots of table locks while old server is doing that in 2 seconds. Zero i/o wait time on both machines, 2-3 load average... he is running 125 agents and wanted something for 250 agents. the new DB server failed. moved him back to old server to see what could be the issue..both vici 8.0.1 although the DB was originally from 7.0.4 when he had only 20 agents. Things happen, not sure what went wrong LOL and yes the table were optimized.

[gequiros]

Do you think the bottleneck may be on the AMD ?

[williamconley]

Do you think the bottleneck may be on the AMD ?

unrelated unless AMD is overloading the server

the new MySQL DB was fetching the same stats in 15s with lots of table locks while old server is doing that in 2 seconds

the same stats is vague. the actual query is necessary to determine where the slowdown is. often it's a missing index.

note that the vicidial db is missing a lot of very useful indexes. why? because if they were all added, covering every possible overload/use scenario, the additional indexes would slow the system down more than any of the individual speedups. counterproductive. so when you find a specific query that takes too long to return a result, you attack that query in troubleshooting mode. if necessary, add an index.

but without the query being shown, all the rest of the information is ... noise.

[gequiros]

Sir William from the Vicidial kingdom, what do you think about the following Setup:

PowerEdge R730 Server
2 x Intel® Xeon® E5-2630 v4 2.2GHz,25M Cache,8.0 GT/s QPI,Turbo,HT,10C/20T
4 x 32GB RDIMM, 2400MT/s, Dual Rank, x4 Data Width
RAID 10 for H330/H730/H730P (4-16 HDDs or SSDs in pairs)
PERC H730P RAID Controller, 2GB NV Cache
4 x 600GB 15K RPM SAS 12Gbps 2.5in Hot-plug Hard Drive,3.5in HYB CARR
Broadcom 5720 QP 1Gb Network Interface Card
Broadcom 5719 QP 1Gb Network Interface Card
iDRAC8, Enterprise with OpenManage Essentials,Server ConfigMgmt
Internal SD Module with 1x 64GB SD Card / This is for DB BackUPs


This beast price at $10k

Will it be handling easily the load ? ( I know, vague question, vague stats and so on, but just to figure it out )

[williamconley]

RAID technicals are interesting, but it comes down to throughput. Huge arrays are cool, but they have to pass all that data through the controller(s). Some controllers overheat, thus making the huge array pointless. So ... load it for bear. Load test your Vicidial cluster to simulate twice as many live agents as you intend to have. If it doesn't overload, you're good to go. If it does, then it's a question of How it overloaded (overheating? slow queries? bad behavior?). Note that you should have at least one real agent logged in running test calls at the same time so a human can detect bad behavior if it happens. The test calls won't do that.

Next up we have hardware specs: But the basic "core count and speed" is not visible. I could check the ARK of course, but I'm lazy. More cores = more power. Hyperthreading doesn't fool linux. So either turn it off OR test the overload and see if the overload happens at a higher or lower level with hyperthreading on/off. Make your own decision.

I will say that you will also need to be sure none of this hardware is too new. OpenSuSE drivers must be available. Bleeding edge hardware may not yet have drivers throughout the Linux continuum.

I'm not sure why a "Server ConfigMgmt" module is needed for DB backups. Ordinarily you have three methods available:
* Vicidial's backup script provides mysqldump full db backup copies (which can be FTP'd off the server with the same script). It can also just be stored on a 2nd HD instead of the primary drive. This allows reformatting the primary drive without losing the backup data
* Binary Logging which provides "up to the moment of death" data restoration, but adds load to the server.
* Replication on a different server. Vicidial also has settings to use the replication server for Reporting which can seriously reduce load on the Database Server.

At $10k I'd probably be looking at two server with 20+ cores at 2.6Ghz and RAID10 with 15K SAS drives (although we do like SSDs, we just don't use them as SAS seems quite solid).

[mflorell]

Here is my often repeated message on DB servers and RAID cards:

For higher-volume vicidial systems, we only recommend using LSI Logic MegaRAID caching raid controller cards. Dell PERC cards are NOT capable of handling the same capacity as MegaRAID cards can.

We typically use SSD drives now with LSI Logic MegaRAID cards, and have installed dozens of servers with these over the last few years.

[gequiros]

Then, based on suggestions, back to eBay hunting for a good Server... lol..

William... that was a total of 20 real cores ( not HyperThreading)

Matt, yes, i read you typing several times for that Raid Card, server will came with that

Think i will get better behavior having somebody build server for me with the specs given by you guys

Will AMD Epyc processors behave as good as XEON ? have any experience on that for DB server ?

You got not idea how much valuable info i have got from all of your post... i keep just asking to have a better understanding

On a second thought ( totally unrelated ), i may be attending IT Expo in FL in a few days, if so, will let you know, beers will be on me... i am having delays on relocating my Call Center on a new spot ( Initially was able to host 100 agents ) now, it will be down to 50, but having "multi-branches" for service and agents redundancy ( Different countries and Cities ) so, it will allow me to "grow" more and as well more "steady"

Thanks so much for always guiding me on the right path...

I will keep adding more more questions to the table...

[williamconley]

Processor brand isn't generally a factor. Speed and core count is. Chipset on the motherboard will determine driver availability. Motherboard brand speaks to quality and likelihood of failure (although all brands eventually die: some makers have better reputations than others for a reason). Also, personal experience tends to lean in different directions for different people. For instance: The Vicidial Group uses SuperMicro servers. PoundTeam NEVER uses SuperMicro because all of the ServerMicro servers we've ever had (to date, only about six) have Died horrible, miserable deaths. Clients who have SuperMicro see about 33% death rate. So we avoid them like the plague (although I do admit that those servers that have died while under warranty have been replaced or repaired within two weeks ... that's a long time to be "down".)

RAID cards can be added to servers as long as there's an expansion slot of the correct size. It's not unusual to ignore a built-in RAID card that is substandard and put in your own powerful card.

[gequiros]

What hardware will you suggest me?

Which Version of LSI Logic MegaRAID card as well which Storage drive will suggest me ?

I will be going for 36 cores or so ( real cores, not threads ) and 128gb DDR4 (this will be the minimum i will get)

If you see something like a good price on eBay that you can suggest me that will behave nice on me, let me know...

[mflorell]

The specific MegaRAID card you want will depend on what kind of server and drives you have:
https://www.broadcom.com/products/stora ... #tab-span3

[gequiros]

What do you think about this DB Server?

2 x Intel® Xeon® Processor E5-2679 v4
Each one:
# of Cores: 20
# of Threads: 40
Processor Base Frequency: 2.50 GHz
Max Turbo Frequency: 3.30 GHz
Cache: 50 MB
Bus Speed: 9.6 GT/s QPI

Total of 40 Cores / 80 Threads

128GB DDR4 2400MHz RDIMM ECC

4 x 480GB NVMe
1 x MEGARAID 9440-8I / Raid 10 ( I guess it will work with the drives )

===============================

After that, i will use a Slave DB Server, of course, will be super ordinary server, nothing powerful

Do you think i can easily think about 300 agents with a beast like that ? how many ( if lucky, or how hard ) you think i can push it ( how many agents it could handle )

Thanks so much... seriously, thanks so much for all the help !!!

[frequency]

That would work easily. Try a normal samsung or crucial SSD for booting the server, mount the nvmeraid to /var/mysql/data if using 8.0.1.,
Slave DB will need RAID card with ssd too, any samsung or crucial DC SSD would do the job. make sure its a DC spec'd SSD and not desktop SSD. it has to keep up with the writes since it would be making 85-90% writes unlike your normal DB server that would be making 80% reads/20% writes.

[gequiros]

Abusing of your time

Can you please let me know, if you will have the call, which drives will you put there ? Which ones for the RAID and which ones for the OS on both servers and which model of LSI MEGARAID will you use there

Thanks so much

Now, everything is boiling down on time to me... everything gets on suggestions where i needed to be....

[gequiros]

Well, looks like i will get that system described above...

And i need to buy a Slave DB, but i don't want there to be a bottle neck or any issue to be caused by SLAVE

What if the Slave DB is NOT as powerful as the main one? can it create a bottleneck ?

Instead of 4 x NVMe RAID 10 (On main), i will use 4 x SSD / 15k SAS drives ( don't think this will be a bottle neck )
Instead of 128gb DDR4 ( main Db) , i will use 64gb DDR3 on Slave
Instead of 40 cores @ 2.5ghz ( as main ) i am thinking on 20 Cores E5-2680 v2 @ 2.80GHz

I know, it will be on real life where we discover any issues, but, can that create bottle necks of any sorts ?

Merci beaucoup ... AMIGOS

[williamconley]

Slave DB power has no effect on the master. The master will write the transfer binaries and then transport them to the slave which will process them independently of the master.

Note that you should always empty all memory tables before rebooting to avoid replication failure. Some systems will balk when the reboot results in a data set that does not match because of the empty memory tables, stopping replication and making it appear as if there is a real problem. Either ignore memory tables or ignore errors from memory tables or be sure to wipe them before reboots (or all the above).

[gequiros]

with that main DB machine i am about to get, do i need to think about getting a slave one ?

[williamconley]

That's based on your usage. If you NEVER experience any load issues, you would only need replication for "live backup set" purposes.

ie: if the HD upon which the DB and mysql binary logs resides dies ... the replication server will have all your data. Up to the second of failure in many cases.

[gequiros]

and for replication, just decent hard drive? what can you recommend me ?

I guess it can be a single drive and nothing expensive, correct ?

16gb DDR3
8 cores
1 x SSD

And i should be good to go, correct ?

[williamconley]

Only relevant if you intend to use it for Reporting (not a bad idea) and when you consider that if your primary DB dies (which should be next to impossible, but still ...), this server will then become your Primary temporarily. At that point you may just be happy to have a functional DB and not care. Or you may want to be able to run full blast even while awaiting repairs to the primary DB. Or somewhere in between.

Translation: That's on YOU bud.

[dhijrwn]

Hi guys, just needing some advice about our cluster setup specs, estimated it to handle 150 agents to 300 agents, blended calling.

DEDICATED WEB SERVER and TELEPHONY SERVER same specs
i7-9700 cpu @ 3.00ghz
gigabyte z390 M
2pcs 1tb hdd raid mirror
2pcs 16gb hyper x2666 ddr4

DATABASE SERVER
i7-9700 cpu @ 3.00ghz
gigabyte z390 M
2pcs samsung 500 ssd
4pcs samsung 250 ssd
2pcs 16gbhyperx 2666 ddr4

ARCHIVE SERVER
i7-9700 cpu @ 3.00ghz
gigabyte z390 M
2pcs 2tb hdd raid mirror
2pcs 16gbhyperx 2666 ddr4

Thank you for your help.

[mflorell]

How many telephony servers?

How many telco lines will your carrier allow?

What line-to-agent ratio do you plan to dial at?

[dhijrwn]

How many telephony servers?

How many telco lines will your carrier allow?

What line-to-agent ratio do you plan to dial at?

currently I have setup 2 telephony servers that can balance dialing, if still neded can add more,

We got 2 voip sip trunk carrier, 1 currently support 50 sessions/telco lines? the other one they said it is unlimited and planning to acquire one or more.

if we have many agents in one campaign like 30 to 40agents i usually do ratio 1:3 then if 10-15 agent in campaign i choose predictive dialing with maximum dial level of 7.0

Thank you.

[mflorell]

If you are going up to 300 agents you will need more telco servers. At a 7:1 dial ratio I'd say you shouldn't put more than 25 agents on a single one of those servers.

As for the telco, there really is no such thing as an "unlimited" carrier, there are ALWAYS limits

As for the DB server, does it have a caching RAID controller card?

If properly configured, and with a MegaRAID card, you shouldn't have any problem with that DB server, as long as you keep your logs archived frequently and keep your vicidial_list table under 10 million leads.

[dhijrwn]

If you are going up to 300 agents you will need more telco servers. At a 7:1 dial ratio I'd say you shouldn't put more than 25 agents on a single one of those servers.

I see, because of the heavy load is on the active calls ringing and session right?
Lets say that I cannot setup the load balancing in the documentations LOAD_BALANCING.txt it is too advanced for me, what I should do is register 25 phone number agents on server a? then register phone number on server b? then when they login first 25 agent will used on zoiper setting dialer ip is server a, after that the other 25 agents they will use server b ip on zoiper phone, this is to seperate and limit it. This is what I know.

As for the telco, there really is no such thing as an "unlimited" carrier, there are ALWAYS limits

haha yes but that what they said.

As for the DB server, does it have a caching RAID controller card?

Since I am not familiar with raid tech, and first time using it on built in mother board but I'll check and ask.

If properly configured, and with a MegaRAID card, you shouldn't have any problem with that DB server, as long as you keep your logs archived frequently and keep your vicidial_list table under 10 million leads.

I think we dont have megarid card used on motherboard.
logs means all the calls reporting/recording in vicidial? if yes then our asterisk servers successfully transferring all the call recording to archive server.

Thank you for your advice and help

[williamconley]

Dialer load based on number of agents: Since you're already in production, you are the best judge of this. Note that 50% usage is the max target for "sleeping well knowing the servers shouldn't die tomorrow". By 50% load, I mean if the server has 8 cores, the average server load should not sustain above 4.0. Peaking above 4.0 from time to time, fine, but if it stays above 4.0 you should look at adding another server soon because much more load may cause overload.

no such thing as an "unlimited"

Some carriers who can only handle a couple hundred new lines say unlimited. Others can handle several thousand and will not use unlimited, but if your servers can only handle 500, then 2000 capacity is unlimited ... to you. lol. Next time they say unlimited, ask them if it's ok if you fire up 2000 outbound lines at once. If they don't balk at that number, they may be good. But it doesn't matter since ALL carriers "go down" from time to time, so having three installed and tested with full load is the only safe way to run.

... raid tech, and first time using it on built in mother board ...

Um, no. If it's built-in it's not good. If it merely "came with", that's a different story. RAID is not required, but suggested for the moment a HD fails which would otherwise shut you down. However, RAID requires superior cooperation between the drives. If they don't work together perfectly, they will slow down the system and then instead of "protection against failure" you have achieved "bogged down, incapable of handling the load". So: RAID 10, preferably MegaRAID if available. And I'd like to suggest that if it's built-in to the motherboard, it's probably only Mirror and may slow down your system. When you go past 75 agents in the cluster, this will become problematic. Not that big of a deal, though, since it's just the Database that needs to be on the RAID10. Not the OS of the DB server, just the mysql data folder of the DB server. So you can add RAID10 and move the data folder onto the RAID when you become aware that your DB is bogging down due to drive speed. This will likely result in instability, weird problems, and slow queries awaiting HD response. If you're lucky, it will first show as "slow call routing" instead of "calls routing incorrectly", which is much more awkward when it happens.

logs means all the calls reporting/recording in vicidial?

Just the reporting, which is database tables ending in "_log" that get HUGE over time as those logs accumulate. There is an archiving script in crontab -e which can be executed with "--help" for options. If you do not archive your logs, the log tables will eventually grow to a level that slows down your DB server and/or will cause table crashes due to the oversided logs. Best to keep only a few months (or ONE month) of logs in the live tables these days since most useful logs and alternately pull from the "_log_archive" tables anyway, so the data isn't logs it just requires checking an extra checkbox when running a report, but dramatically improves db response time compared to excessive log table counts. And of course, keep your leads under 10M. Not "LIVE/Active" leads: TOTAL leads in the vicidial_list table under 10M. Actually shooting for under 5M is better if you can.

[dhijrwn]

Thank you for the information and guide, It is really important to watch the load of servers and maintain database regularly