Questions regarding swap, partitions and HT.

[bryan.kewl]

I want to install ViciBox Redux x86_64 on a server with specs
Quadcore i7-2600 @ 3.4 GHz with HT
32GB of RAM
2x3TB SATA Drives

I am going to create SOFT RAID-1, so will have total 3TB space. I am not sure about the following
- should I disable Hyper Threading?
- Should I create 4GB swap partition or 32GB swap partition.
- I am thinking of creating separate partitions for /boot , /home & /tmp ? or use a single partition for /?
- I am going to record calls, so can someone tell me where the recording is saved? and should I create a separate partition for it?
- What file-system should I use? ext3?

[DomeDan]

You should tell us how many agents, inbound/outbound, codec and stuff like that too.

Now I'm just imagine a 10 agents outbound:
- I would disable hyper threading because I think the database needs fewer cores with more clock-cycles then more cores with less clock-cycles.
- I would go with 4GB because I would probably never gonna use all 32GB RAM.
- I would use a single partition for / but with that much RAM I would create a ramdisk for /tmp.
- It is saved in /var/spool/asterisk/monitorDONE/ a separate partition would probably be a good idea, but even better is a separate drive.
- Yeah ext3, or ext4 if its available.

this is of course what I think, it might not be optimal for your setup

[bryan.kewl]

the primary usage of the system was as a web-server, but it was't being used. so I took the opportunity to test it for vicidial. actually we are looking for a cluster to handle around 500 simultaneous agents for a 4:1 outbound campaign + around 100 agents for 20:1 ratio on a voice broadcast. so according to the 25 agents per dialer rule we are looking around 35-40 servers. I know vicidial don't scale well vertically. If I can bring the servers count down to 20 or below by pushing huge boxes, it will be easy for me to manage them. I am going to stress test this machine this week.

We are also going to get 2 huge machines for database with
- Hex Core 3930K @ 3.2 GHz with HT
- 64GB RAM
- 2x3TB SATA Drives + 2x240GB SSD using Hardware RAID-1
and I will do testing on them as well to find the suitable machines for the cluster.

My main concern is to bring down the nodes count in the cluster.

OS: I am using CentOS 5.6-64bit because it allows the /boot on RAID1 drive while openSUSE 11.x don't allow that. I don't want HD crash bringing down the node thats why all nodes will have 2x or 4x drives in RAID1.
Partitions: I will be using the legacy grub which restricts partition size to be under 2TB. so I will appreciate if someone can suggest the partitions + their sizes. Right now I have 32GB SWAP, 500MB /boot and 1.99TB /.
Codec: G729 codec will be used mainly because of its low BW requirements.
Recording: We will record most of the call.
Hyper Threading: I know mysql can benefit for multiple core, but most of the times its suggested to disable the HT. What about asterisk + vicidial?

[williamconley]

You will have a challenge making bigger boxes handle lots of outbound agents. Remember that more cpus does not give you a 2nd I/O path or frontside bus. Using SSD drives may help reduce the i/o time, but not to the point of doubling agents. Certainly something you should test, of course.

But while we're on the subject, recovering bandwidth via g729 is nice, but will push your CPUs harder, once again challenging your goal of more agents per box. Same thing with recording all calls. This challenges the larger box with more processors concept as well. Look to have plenty of memory and fast drives to move these recordings around quickly.

Hyperthreading has never had much of a noticeable difference when we've played with it. "Fooling" an OS into believing there are more processors available is probably useful under some circumstances, but when faults begin to occur because those fake processors are not really there ... the system overloads just about the same time as it would with half the processors, but all real. However, certainly something you'll want to test on your own to confirm our results!

Partitioning, however, is a completely different story. You do not want large slow hard drives. You want tiny fast ones instead. Do not store any data on these hard drives. Push all recordings to a non-vicidial based web/ftp server and back that server up reliably. This means that any Vicidial server is basically disposable (in case of death, others can fill in and there is no data loss involved!). Perhaps moving the monster sata drives onto some other machine outside vicidial's system would be a good idea. That would also offload the web service for listening to recordings from the vicidial hardware.

Be sure someone is harvesting/pruning the DB regularly. Massive logs can severely hinder response time once the system is up and running. Pushing these logs off onto an archive DB server (for future reporting) is necessary to maintain speed on the system over time. So have a plant to push those records off into archive tables and eventually push the archive tables to another machine entirely for permanent storage (or just delete them).

Consider, also, using a replicated DB for reporting purposes to offload any reporting functions from the Live DB server in Vicidial. But remember that you can only have one Vicidial DB which must be the replication master if replicating. you cannot use the replication slaves for anything beyond reporting, they cannot control dialers because the data usage in vicidial is much too fast. These are not banking transactions, they are instantaneous hi-res timing events that occur at the nanosecond level. No way replication or a mysql cluster can keep up with that. ONE db server per cluster.

that is not to say it would be impossible to build a multi-cluster system that shared information from a central DB, if you properly segregate equipment and personnel. that's actually quite possible.

[Kumba]

Just use 500-GB enterprise sata drives in the dialer and web server. We are partial to the Seagate ES and Wester Digital RE series of drives. The dialer and web server does not need 3TB of storage and it's a bad use of resources. You should have an archive server with a RAID-6 that gets the 3TB drives. The database should get 15K SAS or SSDs configured in a RAID-10.

Not sure where you are getting 50 servers. Here's my math for 500 agents using the general provisioning guidelines:
20 telephony servers with 25 agents per server
4 web servers with 125 agents per server, plus one additional server to act as the front-end load balancer and admin/API server.
2 large database servers, one primary and one slave/backup.
1 Archive server, probably somewhere in the 10+ TB range

By chassis type, this is what you need:
25 telephone/web servers (Quad-Core, 8-GB ECC Ram, 500GB RAID-1)
2 large database servers (Two Quad-Core (or more), 64-GB ECC RAM, SSD in RAID-10)
1 Archive Server (Can be anything you want, but we usually provision them as quad-core, 4/8-GB RAM, and 6+ drives in RAID-6)

27 vicidial-specific servers in total, or 28 if you don't already have something else to use as the archive server. Only need FTP and HTTP for something to act as the archive server. Most NAS solutions or other basic storage servers will do this natively to begin with and save you the cost and hassle of have another server laying around. You can also consolidate further by having the slave database act as the front-end balancer and agent/API interface if you really want to, but with a cluster this size what is a few more bucks for a dedicated machine that you will use.

So, 26 to 28 servers depending upon how you want to provision, and that is at the recommended dimensioning guidelines. If you want to push the deployment per server, with the understanding that you might have some loading issues at some point, you could re-dimension the cluster with 30 agents per telephony server and 175 agents per web server. That would bring your telephony server count down to 17 servers and your web server count down to 3 plus the balancer. Total server count, not counting the archive, would be 23 servers. Please keep in mind that this is what we would consider the safe upper limit. Running above this level really start to become case specific to how you are using vicidial, options you have enabled in the agent's web interface, how much API usage you are doing, dial level, answer rate, lead quality, talk time, etc.

The newer hyper-threading in the last few years is useful for the apache process. With MySQL and Asterisk it doesn't give you much within ViciDial. Feel free to google about it all you want but it will be very dependant upon the workload being performed. ViciDial is mostly IO bound on the database which hyperthreading doesn't help with. Asterisk might show some benefit but more then likely will hit it's own internal software issues before it could fill up 4-cores of CPU anyways. That means Asterisk will crash before it maxes out a quad-core CPU.

I also think you are trying to save money in the wrong ways by using desktop components and scaling vertically, specially with the hard-drives. Those are the one component you can virtually count on to fail. There is more to the enterprise drives then just a longer warranty. I would commoditize on the following hardware platform:

Telephony/Web:
Intel E3-1220 (or E3-1230 if you want HT)
8-GB ECC Ram
two 500GB enterprise-sata drives 7200-rpm
Reputable chassis/power-supply

Big Database:
Dual Intel E5-2609 Quad-Core CPU (or dual E5-2620 if you want Octal)
32-GB ECC Ram (Optionally 64 since RAM is cheap and fast performance for MySQL)
Four 240-GB Intel 520 SSDs in software RAID-10 (optionally an SSD-optimized hardware care) or 8 300-GB 15K SAS drives in RAID-10 on a hardware-backed caching controller
Reputable chassis/power-supply

[williamconley]

I love it when Kumba kicks in with a precise enterprise level build-out. But then again, that is what he gets paid for and deals with daily. I bet that was even fun to write.