ViciBox v.4.0 to head in a different direction

[Kumba]

I've decided to change direction with ViciBox v.4.0. This is due to frustrations or concerns with the following:

1) Organically grown and limited installer
2) Lack of a cohesive cluster installation method
3) Too much prior knowledge needed for current installation methods
4) No real active server restoration method or addition

Because of that, I've changed methods for how ViciBox v.4.0 will work. Fortunately the release was only ever a Beta and so I still have discretion on changing it. To those using it in production that's fine. It will just be like an upgraded ViciBox v.3 install.

What my new goals for ViciBox v.4.0 is as follows:
1) A self-describing cluster installation and tracking method with optional expert mode
2) Ability to add a server to a cluster with minimal user input
3) Slave DB support including configuring and sync'ing itself
4) A '--restore' flag added to the installer to allow ViciBox to reconfigure itself as a failed server


To accomplish this, I am splitting the vicibox-install into two parts. One part will be a PERL script to interact with databases, SVN, and other logic tasks that either aren't obtainable with bash or are not trivial. The second part will be a big long nasty BASH script that cycles through the commands on the CLI and makes things work. The ISO has also been streamlined with the graphical boot screens and some excessive options removed like the network settings and EULA agreement on boot.

As part of accomplishing the new loftier goals for ViciBox v.4.0, some methods used and developed in house for our hosted offering will be deployed in a limited fashion. Most notably is that there will be a table called 'vicibox' created in the vicidial database by the installer. This table will be an internal tracking mechanism used by the PERL piece of the installer to know what settings to inherit from the cluster. Things like Archive settings, DB settings, SVN Version, whether it is the first Telephony server or not, etc. This is all a means of accomplishing the goals mentioned. User input on a compatible ViciBox v.4.0 install should be limited to Database IP, DB Name, User, and Pass. The installer should then be able to pull in all other relevant settings along with the user selected roles to complete the installation. There will also be an expert mode available which follows the same vicibox-install method used in V.3 for backwards compatibility and those who just like seeing the options.

SO, in short, sorry for the delay. I'm actively working on ViciBox again now that I have some of the bugs preventing it from working out of the way. Things are progressing along smoothly in the land of PERL although some things were far easier in BASH. I hope to have a Beta2 available in a week or so. I already have the vicibox-express set-up working but since that doesn't do anything needed for a cluster it was easy. As a side note, the vicibox-express installation is cluster compatible although for obvious reasons you would be better off moving the database over to a new cluster.

Thanks for the input from the other thread. Your feedback is what prompted this change of direction.

[williamconley]

Sounds very intense. I like the --restore concept.

[ZoVoS]

Any progress chaps? I have a few servers in the cloud that I have recently un-provisioned that still have some time left on their rental. If you need any testing I'm more than happy to throw up a few clusters and bounce traffic back and forth to make sure everything is working nicely.

[Kumba]

The installer is progressing along nicely. The --restore option is planned but will probably come as part of a later revision.

As part of a design decision would anyone miss not having the branch available for selection? In other words, I've built the installer around working with SVN trunk. There is currently no option to select installing the 2.4 Branch of ViciDial. You can, however, tell it what SVN version to install.

I am not sure how much it will be missed considering you would have to do a manual or SVN upgrade of an older ViciBox in order to become compatible with v.4 anyways. The other option is to tell V.4 what SVN version to use that corresponds to when the branch was done.

I am running through all the iterations of the installer to make sure things work as expected.

[Trying]

I believe as long as you can tell it what SVN version to install it should be fine for almost all users.

[ZoVoS]

hello again, good to hear that the install is coming along well. just a few things. I made a major cock up today when installing. I believe I got the password for the SQL wrong,

usually its DB -> WEB -> First telephony server.

After rerunning the script I found it still would not insert the correct information into the database server.
rather than a re-install I had a look at the script. I found it was searching for /usr/src/astguiclient/conf/.vicidbinsert
I simply removed the file and it worked fine the second run through.
I was thinking the script should probably prevent the files creation unless the database content is verified.

I was going to make the changes but i assume they will be superseded by v4? as in everything will be stored in a database anyway?

I look forward to reading it over when its done.

p.s. I generally agree with -Trying- regarding the SVN options

edit - Could even have it search the database for the valid information and if its not there -> create it

[Kumba]

The V.4 PERL-based installer verified that it can connect to the installer and also looks for the 'vicibox' table which contains the installer and cluster information. If it can't connect to the database during set-up it dies and lets you know. This in essence makes it impossible to do an install without database connectivity.

It also looks in the 'servers' table to see if an entry already exists. If it does then it does not insert the DB stuff.

In it's simplest form, you install the database, tell it to use the default user credentials, then on the following server you simple tell it the DB IP and to connect using the defaults and it sucks everything in to configure itself. About the only order of install is DB first, as always, and archive server if you want the dialers to pull in those settings.

[ZoVoS]

Another random thought I had was offering a random password (instead of 1234) automatically generated by the installer for all the passwords (i.e. cron password [RANDOM], press return and it would ask you for "How many characters [32]" and then generate a random alpha numeric password to the desired length), at the end of the install it could create a file with the usernames and passwords, big warning saying "your configuration output has been stored in make sure you delete /blabalbla/blablabla/.password"

This would prevent people with little knowledge of how the system works leaving the password as 1234 and then having remote requests to their MySQL

Also the DB server could be queried and the information for the passwords auto filled in.



At the end of the install a master password could be generated and MD5 hashed. (preventing standard access to the script) A button can then be added to the vicidial interface.

Later to change a cron password etc having this master password could run a script that changes all the areas needed including the MySQL

of the top of my head er;

update mysql
flush privileges
astguiclient.conf
manager.conf
update it in the vicidial interface
and update it on the phones?

Again nothing fancy I just don't remember there being a single unified button for this.

Sorry if this is a bit messily typed. its 4:30am and I'm knackered =]

[Kumba]

No clue what master password you are talking about. That's more of a ViciDial then ViciBox function and would require someone to convince Matt to enable that.

MySQL has a 16 character limit on user credentials. I am already generating a 15-character string and inserting it into the system_settings table for the default server conf and the phone registration password. None of the attacks we have seen on vicidial have been MySQL related. They have all been related to SIP brute-force with a bad password. This results in toll-fraud or calling-card scams to eastern europe and the middle east. At least, that's where they are always calling when we see it. Sometimes you see Jamaica sprinkled in there too.

You can't hash any passwords as ViciDial uses and stores them in clear text. If you forget the cron user password then just open up /etc/astguiclient.conf and there it is. I also do not secure MySQL locally so the sysadmins who are aware can run mysql_secure_install to their hearts content. Any time someone who didn't understand what they were doing changed the MySQL password is usually results in us having to break back into it. It's the same reason we use SSH keys on customer's systems because they change root passwords and forget them.

[ZoVoS]

I was thinking a new master password but yes, it would require some modifications to vicidial. I use it my self for making changes based on large scrips to automate stuff. I didn't want to hook it to any existing passwords as I use it to administer servers in the cloud I rent to people (its the only password i don't give out, it hashes the inputted password and checks it against the password in the database before the script can run)

I have seen a few attacks on MySQL (only a few tbh) I run quite a few servers (20-30 servers/clusters) and am constantly monitoring, I was unaware MySQL limited to 16, its quite lucky I only ever generate 16 character passwords, that's something I really should have been aware of (thanks for bringing it to my attention) but my background is in computer networking we usually have somebody else handle the database side of things.

I have made some sweeping changes to secure a few things my side as when I started meddling a year ago I had no idea how anything worked.

Part of my security audit a few months ago I did a simple Google search for vicidial box's and found a surprising number who had left the cron password as 1234 (and had phpmyadmin installed etc.) so i was able to gain full access to their database (don't worry I emailed all parties involved and suggested some changes they should make) only the changes required were a bit complicated for some people to implement (basicaly whoever set there server up just left default stuff and in most cases charged them an extortionate amount of money)

We are actually a telecoms provider and we supply vici servers at rental cost and make our money from the calls.

I was also thinking of adding some simple WEBMIN like interface for handling IP tables etc, and an automated install process for failtoban. Its all part of the modular changes i'm currently in the process of implementing.

edit - we don't give people root access to the box. The changes I have made basically allow people to make most changes without ever touching the internals of the server, I find they mess less stuff up that way. so they will never really have access to the files. The master password is just a very long password that I can give to my support staff to make sweeping changes without them needing access to the root account.

[ZoVoS]

Out of interest would you consider adding an optional failtoban installation into the vicibox install? (if not I can edit your ISO anyway, i was just wondering if that was a direction you wanted to go down?)

[Kumba]

Out of interest would you consider adding an optional failtoban installation into the vicibox install? (if not I can edit your ISO anyway, i was just wondering if that was a direction you wanted to go down?)

The fail2ban package as well as GeoIP xtables package are added in ViciBox v.4. I have a few things on my feature list like implementing a "ban known problem countries" option directly in IP Tables to prevent brute-force attacks from China as well as a fail2ban log scanner to find the other ones. At this point it will have to come as part of a later release since they have finally killed the 11.3 repo since OpenSuSE v.12.2 is scheduled to finally release on Sept 5th. I have the PERL installer written and a rough test done on it. It works but then I found a DAHDI issue in the repo. That combined with 11.3 dieing yesterday prompted me to go through and clean things up. The new software stack looks to be as follows:

OpenSuSE v.12.1
Asterisk 1.4.44-vici
Dahdi v.2.6.1
Wanpipe v.3.5.27.4 (might look at v.3.5.28 if it seems an easy upgrade)
LibPRI v.1.4.12
Wanpipe-Voicetime v.1.0.14
VoiceSync v.1.2.5
OpenR2 v.1.3.2
libSS7 v.1.0.2 (Not sure who needs it, but they can quit bugging me about it now)

It looks like I got the DAHDI issue figured out now and all the software compiling so I just need to run through my cluster tests again and then I can put an RC1 into everyone's hands to give me feedback. Once the RC1 is pushed out then I will start the documentation. As always, feel free to submit any patches to the installer or other code you want included to me. I have no issues sharing the development load with those who are interested.

[ZoVoS]

That is great to know. Id like to thank you on behalf of all the vicibox users everywhere ^_^

[Kumba]

Ok, here it is. Give it a shot. I still have to do some load testing to make sure that MySQL issue in the ealier version isn't there as well as interop testing with other vicibox installs.

http://download.vicidial.com/iso/vicibo ... .beta3.iso
http://download.vicidial.com/iso/vicibo ... .beta3.md5

Noteworthy options:

--vicibox-express = do a vicibox-express install
--legacy = enable legacy mode for non ViciBox v.4+ installs (it will auto-detect this too)
--debug = Debug mode, aka, tons and tons of output that will mean nothing to the less savvy


The actual install file is called vicibox-install.pl. There is a symlink for vicibox-install and a small bash file which feeds it the --vicibox-express flag. Enabling expert mode should get you all the new features like MySQL Slave (yet to be tested) and other install options.

Feel free to download it and test it out! Need to know what bugs you find.

[ZoVoS]

Hey Kumba I have a heavy day of CRM integrations today so not much time to test my self, but a friend of mine was switching over from go autodial and I suggested he gives it a try.
He tried to install twice and each time the root password after the OS install would not work.

After poking around I noticed it was set to blank. I'm not sure what caused his issue (it could be of his own making). I'll probably attempt a install later and check it my self. Just a heads up.

[Kumba]

Hey Kumba I have a heavy day of CRM integrations today so not much time to test my self, but a friend of mine was switching over from go autodial and I suggested he gives it a try.
He tried to install twice and each time the root password after the OS install would not work.

After poking around I noticed it was set to blank. I'm not sure what caused his issue (it could be of his own making). I'll probably attempt a install later and check it my self. Just a heads up.

That's a new one to me. I've done a lot of testing and not once have I had a blank root password. I'm almost more interested in how he accomplished that then if I can fix it.

[Kumba]

And of course I found some bugs in the multi-server install. Already working through those fixes. Other bugs and points of interest are:

1) make SURE you give each machine a unique hostname before installing. This is used by the installer to identify the machines in vicidial
2) The default partitioning scheme is not creating a 4GB swap partition for whatever reason, currently working on this so double-check for it!
3) If you press F3 when booting off the liveCD and select "Text Mode", it will default to feeding the kernel nomodeset which makes it friendly to all video cards
3) The old release does not properly do a multi-server install, but it will work for a vicibox-express install. A new beta will be out soon to address this
4) For some reason after the install and during the first boot, the yast configuration screen sets the terminal in a funkymode. This is a yast bug, just reboot a second time and you are good to go. More annoying then anything
5) Still have to test the MySQL Slave and Archive server set-up, but those are secondary priorities compared to the big three.

Will reply when I have the next beta up. Should be in a few hours I hope.

[Kumba]

Ok, new beta4 is up.

When you get to the partition screen, it will make sure there is a 4GB swap and whatever left for root, but if it finds another swap partition from an old system it will use that too. Everything I tried to do would not make it prevent that. So, you might need to double-check the partitions when installing.

I also added the hostname configuration as part of the first-boot after installation. This should make it easier to set a unique name per server. The other multi-server stuff should be good to go now. Also fixed a few other bugs like enabling the archive on a dialer, etc.

http://download.vicidial.com/iso/vicibo ... .beta4.iso
http://download.vicidial.com/iso/vicibo ... .beta4.md5

[Kumba]

The goods news is that the installer seems to be done. The bad news is that hardware testing revealed the drivers aren't quite production ready yet. Waiting to hear back from the vendors so I can ship a fully-functioning distro.

Current bugs/issues that are inherited from OpenSuSE v.12.1 in general:
1) If yast comes up during the boot process, it turns the screen fonts to garbage. A simple reboot fixes it. Nothing is wrong, just annoying and inconvenient. This is a bug in OpenSuSE v.12.1.
2) The Linux v.3.1 kernel has issues with some ACPI bios'. We found this to exist on SuperMicro X7SBL-LN2 motherboards so far. Disabling ACPI in the bios fixes it. No appreciable effects were noted. This is a Kernel bug not OpenSuSE. Kernel v.3.2+ doesn't have this issue.
3) When you do an os-install, there is no setting available to force it to wipe out the partition table before calculating the partition layout. If it finds a linux swap partition, it will keep it, then create another 4GB partition and use them both. Simply doing a 'dd if=/dev/zero of=/dev/sdX bs=512 count=1' will wipe out the partition table and give you the correct partitioning. Alternatively, you can just use the partitioner during the live-install as well. There is no drawback here other then some minor loss of disk space for the second swap partition.
4) There is no way for me to force text mode during install or from the CD boot screen. The settings used when booting from the CD are passed to the installed system. To maintain maximum compatibility with VGA cards, simply press F3 during the CD boot screen and select "Text Mode". The 'nomodeset' option is automatically passed to the installer but is overridden if a vga mode (vga=) is present.


Current ViciBox specific bugs/issues:
1) VoiceSync and Wanpipe-Voicetime driver are broken. They currently cause a kernel panic in the dahdi kernel module which freezes it up. Working with Sangoma and Amfletec to resolve.
2) The wanpipe drivers are complaining about "wanpipe_lip" module not being found. They never complained before but DAHDI v.2.6 seems to have broken all kinds of things. Working with Sangoma to resolve.
3) Need more iterate testing but getting pressed for time. Don't want to leave the vicibox community in a lurch for too much longer since OpenSuSE v.11.3 is officially dead dead.


The good news is I figured out a way to continue building OpenSuSE v.11.3 packages. The bad news is that they use the updated driver packages which means the VoiceSync, VoiceTime, and Wanpipe drivers are broken at the moment.

Ahhh, if only I didn't have to deal with drivers. Joy.

[bryan.kewl]

any ETA for the final release?

[Kumba]

Looks like I am going to have to push it out with potentially broken drivers for the short term. Amfeltec got back to me with an updated driver fairly quickly, but Sangoma is still in queue. Hopefully it doesn't take too long.

[rrb555]

Hi Kumba,

what installer do you suggest for new server installation? I have new server installation coming in a few days. Will Vicibox 3.1.15 can still do the magic or should I use the updated Vicibox 4 beta installer instead?

Thanks,

[bryan.kewl]

any chances of releasing 64bit vicidial?

[4jc0m0]

@Kumba, is the finished installer included in the beta4 ISO? If not, can you please post it here, or better on the SVN?

Also, are there any plans to add support for other distros than Vicibox? A good starting point for a selection might look like this: Debian, Ubuntu, Red Hat/CentOS/Scientific Linux, Suse. Cheers.

[DomeDan]

@Kumba, is the finished installer included in the beta4 ISO? If not, can you please post it here, or better on the SVN?

Also, are there any plans to add support for other distros than Vicibox? A good starting point for a selection might look like this: Debian, Ubuntu, Red Hat/CentOS/Scientific Linux, Suse. Cheers.

The finished installer will not be called beta

Vicibox is OpenSuSE with vicidial
and vicidial will probably run fine on any distribution.

[4jc0m0]

The finished installer will not be called beta

Thanks, DomeDan. Is it published anywhere yet? Some people might be able to use it as a starting point for new, "from scratch" style installations on distributions other than OpenSuSe.

Vicibox is OpenSuSE with vicidial
and vicidial will probably run fine on any distribution.

I know what Vicibox is

With Kumba's announcement about improving the installer, I was hoping that maybe I can suggest making it (the installer) more distribution-agnostic, that is, extend it to work on several other popular distributions, as it would be hugely helpful for people that have to use a certain distribution due to internal policies in their organization.

There is nothing wrong with having a "preferred" distribution but considering VICIdial is basically a PHP application, extending the installer to support a reasonable list of some of the most widely supported distributions like RHEL/CentOS/Scientific Linux/Fedora, Debian/Ubuntu/Mint and of course vanilla OpenSuSe, dealing with pre-requisites and dependencies and other sanity checks, and maybe with clustering too, would, in effect, turn one of several distributions into a "VICIdial box" and would be a big help for people who are not (yet) very familiar with VICIdial but are otherwise competent at administering Linux-based OS's and networks.

[mflorell]

We will probably never support Red-Hat-family distros(RHEL/Centos/Fedora) with a full vicibox-style installer for a whole lot of reasons, most of which have been mentioned on this forum several times before. In short, red-hat is an unstable mess. We tried to support Ubuntu, but found it too cutting-edge and desktop-centric to be a stable server environment to maintain an installer for. Open-SuSE has the suse studio, which allows easy creation and maintenance of ISO installers over the web. SuSE is one of the most stable Linux distros for high-load systems(like Vicidial), and we have been using it for years with great results. I'm sure if someone was willing to do the work and maintain a separate distro installer, that we would help them a bit, but it takes too many resources to do that on our own, especially when the SuSE Vicidial installer meets all of our needs and is the best option for maintaining an installer.

[bryan.kewl]

does OpenSUSE 12.1 support /boot on Soft RAID-1? OpenSUSE 11.x don't support /boot on soft RAID-1.

[Kumba]

does OpenSUSE 12.1 support /boot on Soft RAID-1? OpenSUSE 11.x don't support /boot on soft RAID-1.

If you mean Linux Software Raid aka the "MD" device/driver then yes, linux has done that for years. I have a couple hundred servers running a software RAID-1 on Opensuse 11.3 and a handful still on 11.1. I assure you it works.

If you mean whatever BIOS RAID you have enabled on your motherboard? No clue, depends on if the BIOS RAID is supported by kernel v.3.1.10. This also applies to fake RAID controllers like a RocketRaid 2310, etc.

True HARDWARE RAID cards like LSI 9260 and Adaptec 6/6T series then yes, they are supported in this kernel. Please check whatever RAID solution you have is going to be compatible with the kernel drives in Linux Kernel v.3.1.10, as that is the current OpenSuSE v.12.1 kernel.




In a somewhat related note, the RC1 of ViciBox v.4.0 is available here: viewtopic.php?f=8&t=25832

[spacejanitor]

Thank you for your continued efforts, Kumba.

Has anybody had experience with the --restore flag yet on a failed server? Would be interested in getting "prepared" for this process BEFORE it happens.

[Trying]

The installer is progressing along nicely. The --restore option is planned but will probably come as part of a later revision.

As part of a design decision would anyone miss not having the branch available for selection? In other words, I've built the installer around working with SVN trunk. There is currently no option to select installing the 2.4 Branch of ViciDial. You can, however, tell it what SVN version to install.

Where can you tell it which version/revision it must install? I installed but did not see any such option?

[Kumba]

You have to say 'Y' to the expert install when you install the database to specify. The only other way it will prompt you is if it detects it's not part of a v.4.0 install, which switches the installer into legacy mode. From there it will ask you what SVN revision to get.

[Trying]

Weirdly enough I did try the expert install mode as well and it did not ask me for the revision. I then did a revision checkout which seemed to work but connection to MySql was lost after rebooting. The Vicidial Group fixed it by installing MySql 5.1. They reckon Vicidial has issues with 5.5.

[Kumba]

Weirdly enough I did try the expert install mode as well and it did not ask me for the revision. I then did a revision checkout which seemed to work but connection to MySql was lost after rebooting. The Vicidial Group fixed it by installing MySql 5.1. They reckon Vicidial has issues with 5.5.

There's a tempermental bug with MySQL v.5.5 and ViciDial for some reason. The issue is that it's not consistent. Sometimes things work great, sometimes they don't. We believe the issue has to do with the use of the 'mysql' extension in PHP and the re-arranged network setup of the newer MySQL. We are looking at converting ViciDial to use the 'mysqli' extension for PHP but it's not a simple substitution. For some bizarre reason the developers of mysqli decided to make mysqli_query have the link as the first argument and the statement as the second. The mysql_query function had the statement first and the link second. Just one of those PHP design decisions that makes you go "HUH?"

The issue does not seem to really be prevalent with an all-in-one machine communicating over the MySQL socket. It only seems to creep up when you have the web server communicating over TCP to MySQL. The PERL side of things is completely unaffected. The behaviour that is indicative of this problem will be all agents will be waiting for a call, and then get a notice that they have been paused. This will happen for almost all agents, not just 1 or 2. The realtime will also show them as being in Dispo even though they aren't when this happens.

The fix is relatively easy but not entirely trivial. First stop MySQL so that it's not running, copy the /etc/init.d/mysql script somewhere safe like /root, download and install the 5.1 RPM package from MySQL's website, do a 'zypper -i <mysql-rpm-file>' to install it, and then copy the mysql init script from /root back to /etc/init.d/mysql. It is also an EXTREMELY good idea to do a full MySQL back-up prior to any of this.

Here's the MySQL v.5.1 version we use:
http://downloads.mysql.com/archives/mys ... x86_64.rpm
http://downloads.mysql.com/archives/mys ... 3.i386.rpm

You should really only perform this database software change if you are experiencing the issue described. Once you do this it will present issues on doing OS updates in the future.

[Trying]

Is it not better then to use 5.1 in the distro's until the issue is resolved?

[Kumba]

Is it not better then to use 5.1 in the distro's until the issue is resolved?

If we had known it was an issue prior to release we would have tried to push it in. The RPM install isn't the best option either, it can cause dependency issues with other packages. We still don't know exactly what the issues with MySQL 5.5+ is, or even if using the mysqli extension will help. It's just something to try at this point. The PHP mysql extension has pretty much been deprecated in php v.5.3+ and mysqli is where all new development efforts have gone. So there may very well be a mysql extension bug but it's not getting fixed.

Just trying to duplicate the behavior is proving pretty difficult. We will create a cluster, put 5 people in it, and over the course of an hour have maybe 1 agent exhibit this kind of behavior. If we had a client that wouldn't mind being down while we poke around in the code and try to see what is going on then we might have a better idea. Usually when a client calls us and we see this behavior we just replace the MySQL binary so they can go back to work.

MySQL 5.5 has all sorts of issues with older legacy code, not just ViciDial. They deprecated and removed a lot of older syntax and compatibility. Joomla was hit by this issue as well, although theirs was mostly in the formatting of the SQL statements.