Leads Loading

[Apriljoey]

Hi,

Anyone have idea how to speed up loading of leads in Vicidial in a more faster way, coz loading huge leads takes so much time, any optimization we can implement to optimized it and make it faster somehow.Thanks

Loading like 500K leads

[mflorell]

How are you loading them now?

[santhuarnold]

Adjust in the php.ini..

Please find the below link for your information..
http://uniquesupport.blogspot.in/2015/0 ... le-in.html

[ruben23]

@Matt,

We are loading in Leads loader and its taking so much time loading, are there any other process to load leads, like 800K-900K leads..? Thanks

[mflorell]

You can try using the CLI lead loader, which you can set up to pull files from an FTP account. It is not as flexible, but it is much faster than the web lead loader.

# /usr/share/astguiclient/VICIDIAL_IN_new_leads_file.pl --help
allowed run time options:
[--quiet] = quiet
[--test] = test
[--version] = version
[--forcegmt] = forces gmt value of column after comments column
[--debug] = debug output
[--format=standard] = ability to define a format, standard is default, formats allowed shown in examples
[--forcelistid=1234] = overrides the listID given in the file with the 1234
[--forcelistfilename] = overrides the listID using last number in filename: (XYZ_1234.txt = list ID 1234)
[--forcephonecode=44] = overrides the phone_code given in the lead with the 44
[--file-prefix-filter=WXYZ] = will only process lead files that begin with the characters you define
[--new-list-for-each-file] = creates a new list for each file loaded, listID = YYYYMMDDX where X is incremented
[--new-listid-prefix=X] = prefix for listID when creating new lists, must be only numbers, and 4 or less digits
[--new-listname-prefix=X] = prefix for list name when creating new lists, will be followed by filename
[--new-list-campaign=X] = campaign that the new list will be assigned to
[--new-list-active=X] = Y or N, if list is to be set as active when created, default N
[--new-list-reset-times=X] = reset times(4 digits each dash separated) default is blank
[--new-list-tz-setting=X] = COUNTRY_AND_AREA_CODE|POSTAL_CODE|NANPA_PREFIX|OWNER_TIME_ZONE_CODE default COUNTRY_AND_AREA_CODE
[--USACAN-prefix-check] = check for the 4th digit 2-9, USA and Canada validation
[--USACAN-areacode-check] = check for the valid phone code 1 areacodes, USA and Canada validation
[--NANPA-ac-prefix-check] = check for the valid areacode and prefix from the NANPA prefix database, if loaded
[--duplicate-check] = checks for the same phone number in the same list id before inserting lead
[--duplicate-campaign-check] = checks for the same phone number in the same campaign before inserting lead
[--duplicate-system-check] = checks for the same phone number in the entire system before inserting lead
[--duplicate-tap-list-check] = checks for the same title/alt-number in the same list ID before inserting lead
[--duplicate-tap-system-check] = checks for the same title/alt-number in the entire system before inserting lead
[--duplicate-tnm-delete] = checks for same phone and title in same list then deletes original if title is different
[--dob-mmddyyy] = checks for slashes in the date of birth field and formats from MM/DD/YYYY to MySQL
[--dob-ddmmyyy] = checks for slashes in the date of birth field and formats from DD/MM/YYYY to MySQL
[--postal-code-gmt] = checks for the time zone based on the postal code given where available
[--time-zone-code-gmt] = checks for the time zone based on the owner field time zone code given where available
[--nanpa-gmt] = checks for the time zone based on the NANPA prefix database, if loaded
[--ftp-pull] = grabs lead files from a remote FTP server, uses REPORTS FTP login information
[--ftp-dir=leads_in] = remote FTP server directory to grab files from, should have a DONE sub-directory
[--email-list=test.com:test2.com] = send email results for each file to these addresses
[--email-sender=vicidial] = sender for the email results
[-h] = this help screen


This script takes in lead files in the following order when they are placed in the /usr/share/astguiclient/LEADS_IN directory to be imported into the vicidial_list table (examples):

standard:
vendor_lead_code|source_id|list_id|phone_code|phone_number|title|first_name|middle|last_name|address1|address2|address3|city|state|province|postal_code|country|gender|date_of_birth|alt_phone|email|security_phrase|COMMENTS|called_count|status|entry_date|multi-alt-entries
3857822|31022|105|01144|1625551212|MRS|B||BURTON|249 MUNDON ROAD|MALDON|ESSEX||||CM9 6PW|UK||||||COMMENTS|2|B|2007-08-09 00:00:00|7275551212_1_work!7275551213_61_sister house!7275551214_44_neighbor

stdrankowner:
vendor_lead_code|source_id|list_id|phone_code|phone_number|title|first_name|middle|last_name|address1|address2|address3|city|state|province|postal_code|country|gender|date_of_birth|alt_phone|email|security_phrase|COMMENTS|called_count|status|entry_date|rank|owner|multi-alt-entries
3857822|31022|105|01144|1625551212|MRS|B||BURTON|249 MUNDON ROAD|MALDON|ESSEX||||CM9 6PW|UK||||||COMMENTS|2|B|2007-08-09 00:00:00|99|6666|7275551212_1_work!7275551213_61_sister house!7275551214_44_neighbor

minicsv:
address1,city,name,phone_number,state,postal_code
"105 Fifth St","Steinhatchee","Frank Smith","3525556601","FL","32359"

minicsv02:
name,address1,city,state,postal_code,phone_number
"Adam Smith","123 Fourth St","Englishtown","NS","B0C1H0","902-555-1212"

fixed254:
fixed width(without the quotes)
"9185551212ROSE SMITHS 155 TIGER MOUNTAIN RD. RR 1 BOX 107 HENRYETTA OK74437-941DEMG 226555 0 "

sctab08:
STATE NAME PHONE FAX ADDRESS ADDRESS2 CITY ZIP
AL Tom Wilson 205 5551212 1689 15th Ave BESSEMER 35020

sccsv11:
STATE,ID,NAME,ADDRESS,ADDRESS2,CITY,COUNTY,PHONE,FAX,ZIP,COMMENTS
AL,16700,Fairfax Retirement Center (Crowder House),1324 25th St,,FAIRFAX,JEFFERSON,(205) 555-0508,,35733,Comments

dccsv10:
VENDOR_ID,FIRST_NAME,LAST_NAME,PHONE_1,PHONE_2,PHONE_3,PHONE_4,PHONE_5,PHONE_6,PHONE_7
"100998","ANGELA ","SMITH ","3145551212","3145551213","3145551214","0","3145551215","3145551216","0",

pipe30tz:
TEST01||09292011|1|5125554727||Mike||Frank|||||||||||||||||C|2145559922|TESTSURVEY|TESTSURVEY|111
vote17csv:
Voter ID,County Code,Name Last,Gender,Race,Birth Date,Party Affiliation,Precinct,Congressional District,House District,Senate District,County Commission District,School Board District,Age Bracket,_2010_GEN,_2012_PRI_A_Date,Phone_10digit
"110306742","DAD","Da Silva Ramos","F","5","06/12/1950","DEM","100","23","0","0","0","0","4","A","","0033559433"

t2r13csv:
first_name,last_name,address3,comments1,province,phone_number,address1,city,state,postal_code,comments2,comments3,comments4
Bob,Smith,"Bob, Inc.",lead,president,3125551212,1234 main st,Chicago,IL,60987,good,1235,T2R

twotab:
UniqueID,PhoneNumber
110306742,3125556666

dccsv43, dccsvref51, dccsv52 and dccsvref52:
---format too confusing to list in the help screen---

[ruben23]

@Matt,

But i have custom field with it, is it still possible to include the custom fields somehow..? Thanks

[mflorell]

Nope, with custom fields you have to either use the web lead-loader or the Non-Agent API.

[williamconley]

@Matt,

But i have custom field with it, is it still possible to include the custom fields somehow..? Thanks

THAT would require custom code. It would be possible to run a "followup" script that would acquire the custom data and put it in the associated custom fields.

Of course, the best method would be an add-on to the existing script that would be capable of custom fields.

If you have a limited scenario (one list that always has the same fields), it would not be as expensive to accomplish that as it would be to completely rewrite the FTP import script, of course.