List loader not loading

[Bentfinger]

I am creating a cluster install using Vici 5.0.3 / Vicibox ISO DL'd and Md5 checked on 03/11/2014.
The cluster consists of:
1 DB server (PowerEdge 2950, 2 Quad core Processors at 3.0GhZ) Internal IP only on Eth0
1 Web server (PowerEdge 2950, 2 Quad core Processors at 3.0GhZ) Internal IP on Eth0, External IP on Eth1
1 Dial server (PowerEdge 1950, 2 Quad core Processors at 3.0GhZ) Internal IP on Eth0, External IP on Eth1

All servers have a unique Hostname, using the same Hostname for all network Interfaces on that server. All domains have been set to vici.local

All servers were set up using os-install command from the ISO, then the vicibox-install command from CLI. The RAID was set up prior to any install. No additional Software Installed.

I also re-setup my single dialer *(specs posted below) with vicidial express. from the same .iso

Both installations seem to have gone well, zypper updates all installed, even the "packman" repository that was missing was re-directed and updated.
On the cluster install, web and dial server are time-synced to my DB server.

Yet, with both installations, I am having problems loading and downloading lists. Antything over 25K records doesn't load. I've tried the old list loader with the same results, and when I load multiple small lists into one larger list on the dialer, I get blank lists returned when I try to download the list.

I have installed vicibox 3x before (twice version 4.X, and once 5.0.3)and never had this problem. Any ideas, or direction on what output I should show here to help discover where the problem is?

Additionally, this problem occurs whether I am using IN IP or EXT IP. In the single server setup, it is programmed with both, Eth0 IN and Eth1 EXT; in the cluster, The web server and dialer are setup the same way, and the DB server has IN IP only, on Eth0

[DomeDan]

Both installations seem to have gone well, zypper updates all installed, even the "packman" repository that was missing was re-directed and updated.

Maybe the upgrade changed some settings to apache or php, compare the config files with those from a vicibox live-cd.

check the logs /var/log/apache2/error_log and access_log for clues

[boybawang]

check php.ini configuration

[Bentfinger]

Both installations seem to have gone well, zypper updates all installed, even the "packman" repository that was missing was re-directed and updated.

Maybe the upgrade changed some settings to apache or php, compare the config files with those from a vicibox live-cd.

check the logs /var/log/apache2/error_log and access_log for clues

Thanks Dome Dan. I found some interesting items in the log:

[Thu Mar 13 20:45:01 2014] [warn] RSA server certificate CommonName (CN) `linux-tbnb' does NOT match server name!?
[Thu Mar 13 20:45:01 2014] [notice] Apache/2.2.22 (Linux/SUSE) mod_ssl/2.2.22 OpenSSL/1.0.1e PHP/5.3.17 configured -- resuming normal operations
[Thu Mar 13 21:16:06 2014] [error] [client 46.8.89.24] invalid request-URI
[Thu Mar 13 21:16:06 2014] [error] [client 46.8.89.24] File does not exist: /srv/www/htdocs/HNAP1, referer: http://
MY EXT IP/
[Thu Mar 13 22:08:50 2014] [error] [client 97.66.248.15] invalid request-URI
[Thu Mar 13 22:08:50 2014] [error] [client 97.66.248.15] File does not exist: /srv/www/htdocs/HNAP1, referer: http://MY EXT IP/
[Fri Mar 14 00:05:41 2014] [error] [client 207.244.66.108] File does not exist: /srv/www/htdocs/vtigercrm
[Fri Mar 14 00:59:47 2014] [error] [client 189.221.218.228] script not found or unable to stat: /srv/www/cgi-bin/php
[Fri Mar 14 00:59:50 2014] [error] [client 189.221.218.228] script not found or unable to stat: /srv/www/cgi-bin/php5

The first item is that linux-tbnb is the name linux assigned to all my machines, I changed those during setup to ViciDB01, ViciWEB01, and AiciDIAL01 but something must've not taken. Should I try to change the RSA server certificate CommonName (CN), ad ; if so; can you direct me to the easiest way to do this?
(I also re-setup my single server dialer and am having the same lead loading problem with it, with the same Apache2 error message. It is the only part of the error message that is replicated in both servers, so my gut says that this is the problem)

The other is that I do not recognize any of the other IP addresses in the log for my multi-server install errors posted above; except for MY EXT IP; I do not have a carrier loaded in yet, and during testing did not have my firewall up. Are these hack attempts/successes. Should I re-setup the entire install, activating the firewall from the very start?

Thanks guys, I am excited to find that error, now I just need to figure out how to fix it! I changed all the server names asap during setup, so I do not why this error is there. I've set my multi-server up 3X already because of this problem. I do not mind doing it again, but I do not want the problem to happen again. I look forward to your insights!

Keith B

[Bentfinger]

My LAN settings for multi-server setup is as follows:
Main Hostname in Hostname/routing page is the server name, ViciDB01 ViciWEB01 and ViciDIAL01. my domain for all is vici.local. The Hostname for ETH0 in the details page (Internal IPs) are the same as above, for ETH1 web and dial only (external IPs), they are blank.

I checked the HOSTNAME file in etc on all servers and the Hostname is correct.

[williamconley]

Yet, with both installations, I am having problems loading and downloading lists. Antything over 25K records doesn't load. I've tried the old list loader with the same results, and when I load multiple small lists into one larger list on the dialer, I get blank lists returned when I try to download the list.

Lan settings and cluster settings do not related to failed list loads.
Usually this turns out to be bad leads/lead format or list too large which causes a timeout in apache.

Sometimes it has been a permission issue or a "failed/poor" customization attempt that merely broke the system.

load the leads directly on the DB server (remove clustering concerns, you can actually turn off all the other servers if you like, to be sure, LOL)

[Bentfinger]

Thanks for the input.
I suspected bad data/lists at first also (I usually figure "user error" first), but no matter how I formatted the data, I kept getting the same result. All file sizes were under 8MB.
I did absolutely no attempted customizations on the install.
All permissions for the Admin acct were set to allow loading and downloading lists in the phpAdmin. Is there somewhere else I should look for permissions?
And... I would love to load the leads directly into the DB, but I do not know how. I can use a CLI, but I am very limited in my SQL knowledge and Maria DB is brand new to me as well.

[williamconley]

no matter how I formatted the data, I kept getting the same result. All file sizes were under 8MB.

You're missing the point here. You should attempt to load a single lead with the standard lead loader method. This is best done with tab delimited leads in the columns expected by the system (skipping the "custom" lead loader script by having these leads pre-formatted in the proper columns). Then the lead conversion never happens in Vicidial, just processing into the database. Plus, you should not be "formatting" a bunch of leads, You should turn off dupe checking and load the same SIMPLE lead (with basic information only, phone_number/phone_code/list_id/country_code) 25k times to see if it will load.

First load it ONCE. After all, if you can't get ONE lead to load, you'll never get 25k regardless of "format". LOL

Then 100, then 1000, then 10k, then 25k of the SAME lead. If at some point it breaks, we have something to chase. But if you continue with your existing data you prove nothing when it does NOT work. It didn't work before, it doesn't work now .. nothing proven.

All permissions for the Admin acct were set to allow loading and downloading lists in the phpAdmin.

The permission I refer to are not related to the vicidial admin screen. If you're on the lead loader page, you already have permission. They were also not in any way related to phpMyAdmin, but the mere fact that you are looking at phpMyAdmin tells me you are quite likely overcomplicating the issue and may well have broken something. Vicidial was built to be used with phpMyAdmin by users who don't even know what MySQL is (could be a beer brand?).

I was referrring to the OS permissions. Which can be damaged by command line access. Copying files/folders. Making folders, changing permissions and users.

And... I would love to load the leads directly into the DB, but I do not know how.

You would NOT enjoy attempting to load them directly. This usually does break vicidial.

BUT: You may enjoy the vicidial command line lead loader. Stored in /usr/share/astguiclient. Perl script. Capable of loading MILLIONS of leads in one shot. Very handy. Strict rules, though.

The columns to load your data into are available by pressing the ( ? ) next to the "browse" button when you are about to choose your lead file. Those columns, in that order, will allow you to load the leads as "standard' instead of "custom" very cleanly and very quickly. Those same fields can also be used to load directly into the command line lead loader.

[adymeblack]

I am having the same issue.

VERSION: 2.8-426a
BUILD: 140214-1643
Asterisk: 1.8.25.0-vici
ISO Install: ViciBox5.x86_64-5.0.3
2x- Intel Xeon Quad-Core CPU
16GB RAM
No additional hardware
No customization
[Please let me know if i am forgetting something]

I was able to load lead lists with no issues last week. Tried to load some today and it just won't take them.

I was able to load one list. When I tried with a CSV file and a custom template, all the fields would like "bad lead file".
Changed the format to tab delimited text file with a custom template, no problem. Try again with a slightly larger file, and it just cycles and resets like i refreshed the page,regardless of format.

It will allow me to add single leads.
My php is set up to accommodate the file size and i found nothing out of the ordinary in the apache logs.

I understand that some OS permissions can be getting in the way, but what could they be and how did they get changed? I have made no customization to this system outside of the GUI so I am puzzled.

Ideas?

[williamconley]

Larger file may be too large. Break it into smaller files.

[Bentfinger]

Thanks again Bill, but I am afraid I didn't explain myself fully for you to think I am missing the point. I can load 1 lead, I can load 100 leads, I can load 25K leads, I just can't load more than that. I checked permissions everywhere, read/wrute on folders and Admin permissions. AND.... I used exact lists that I have loaded into 2 previous installs of Vici! So they did work before, just not now.
I tried different formats, because I was sick of banging my head against a wall.... but no matter how small the file size is when I upload it, 20,000 leads results in a file size of about 4MB when I download the list, so over 40K leads in a list and I get blanks fields when I download it. (I assume this has something to do with the 8MB file size limitation.
These are the exact same lists that I have loaded into Vici 4.X previously and into 5.0.3 that I set up in October. No re-formatting, just a plain .csv file. All the other information I listed (re-formatting, etc) was done after loading theselists failed repeatedly.
I am excited to try out the command line list loader... I will post results.

[adymeblack]

The list i was having problems with was only 10mb. I was able to get it to load by splitting it up into 2 different files.

However i loaded the same exact 10mb list on my v 4 cluster and it took it like a champ with no issues what so ever. My v4 cluster has uploaded 20+mb files and didn't even hesitate.

I may have to follow bentfinger and look into the command line loader see if that works better for me.

[williamconley]

Different revisions of Vicibox have different settings for the maximums for upload.

Same goes for download.

It is also possible to have special characters break either one (although this has been cleaned up fairly well, it is not likely perfect yet).

IF you get your leads reliably the same way every time and IF you have a programmer on staff "capable", it is actually quite economical to invest in building a method to push these files in a uniform fashion to the command line lead loader and auto-load them.

It is entirely possible to store the file on an ftp server after formatting it properly and then create a button on the vicidial web interface to "go get that file" and from there it is fully automated. And then all you need to worry about is formatting it properly before placing the file (and remembering to push the button ...). We have several clients with automated processes that pull leads from other systems (nightly) and auto-load into Vicidial (every morning before shift).

[adymeblack]

I think i'm going look into the programming to push the leads to the server. I'm not the best programmer, but the best way to learn is to give it a go [on a machine not in production of course]

I still am somewhat confused about it though. After i made that post, i tried loading the same 10mb list to another v5 cluster i have in a CSV format and it loaded with no issues. I've got 2 v5 clusters that just seem to keep giving me issues, different issues, but issues nonetheless.

[Bentfinger]

It is a confusing issue, but I'll be taking the same approach as adymeblack, I have 2 installations currently..I had planned on cleaning out the data from the single installation and adding that dialer to the cluster, but I may just purchase another server for that purpose, leaving 2 installations available for me at all times.
My impression is that the update packages installed during setup caused the difference; it is the first thing that comes to mind when I re-installed Vici 5.0.3 on the same equipment, using the same .iso, within a few months, but have different results loading leads once installed. Everything about the set-up was exactly the same.

[williamconley]

It's actually very likely it's just a matter of adjusting php to accept larger uploads. Likely that opensuse modified the base preferences and kumba didn't override them when building vicibox. We've reset this on a number of platforms for clients.

That being said, if you're going to "tackle it", it may work best (in the long run) to make sure your leads are stored in the "Standard" columns (thus avoiding the custom lead loader entirely) and this will also allow you to set up a simplistic pathway to "ftp in" the files and auto-load them. You can start by telling the command line lead loader (at the command line) where to grab the file via ftp and how to load it (create a list or add it to an existing list, that sort of thing). After you perfect that, it's not a huge step to put that process in a button or on a web form. Then you'll be able to just push "go" or put in a filename and then push "go". No coding necessary. And it'll still work after an upgrade! It would even work on future "fresh server builds" regardless of what opensuse does to the preferences next time.

[adymeblack]

I've set the PHP in apache to accept that file size and more but to no avail. Is there somewhere else that i need to set it that i may have forgotten about?

[williamconley]

post the precise settings you changed. then we may be able to tell you if you missed any. since we don't know what you changed ...

[geoff3dmg]

There's various interdependant settings you need to tweak. The basic PHP settings are:

Code: Select all

post_max_size 200M
upload_max_filesize 200M

If you increase the file size limits, memory usage starts to become a problem.

Code: Select all

memory_limit 300M

Beyond this, if the uploader has a slow connection, the time limit can become an issue.

Code: Select all

max_execution_time 259200
max_input_time 259200

So in summary you end up with these php.ini settings:

Code: Select all

post_max_size 200M
upload_max_filesize 200M
memory_limit 300M
max_execution_time 259200
max_input_time 259200

Although personally I wouldn't recommend trying to upload 200mb CSV files via the lead loader web page in the first place.

[adymeblack]

I changed the same settings. The numbers are a bit lower though.

[williamconley]

I changed the same settings. The numbers are a bit lower though.

raise them? LOL

[adymeblack]

I raised them from the original values, just not as high as geoff3dmg did. Upload max of 25m instead of 200.