Email

[mister511]

Good afternoon colleagues
When receiving mail and if there are attachments, the text is not displayed correctly

https://ibb.co/ZSC4bJz

[mflorell]

Check the default encoding(or charset) on your database, as well as Apache. Should probably be UTF8 if you are using a non-Latin alphabet.

[mister511]

Check the default encoding(or charset) on your database, as well as Apache. Should probably be UTF8 if you are using a non-Latin alphabet.

If sent without attachments, just text. Things are good

[mflorell]

If possible, can you CC one of your own email addresses with a test email and then post the raw email text you receive on this malformed email from that other email box?

[mister511]

If possible, can you CC one of your own email addresses with a test email and then post the raw email text you receive on this malformed email from that other email box?

Когда войдёшь в наш старый дом:
без потолка, обоев, лампы,
без фотографии с котом,
лишь с нарисованным окном,
где всё осталось на потом,
из красок виртуальных ампул.

Возьми, и, жизнь перелистни
на пару лет страниц Христовых,
где память чертит сетью дни,
где были б мы с тобой одни,
с безродной правдою сродни,
с блокнотом глупых зарисовок.

Зайди туда, где всё сбылось,
где нет углов из многоточий,
где в стену вбит наш первый гвоздь,
чтоб всё взаправду, а не вкось,
чтоб это всё не порвалось,
повиснув нитью одиночеств.

Возьми и дорисуй контраст,
живого платья с цветом клёна,
своих горящих счастьем глаз,
интимных стрелок поздний час,
где не осталось всё без нас
стеклянной пылью на иконах.

Войди в тот дом, чтоб он дышал,
пусть будет он и нарисован.
Жить в отражении зеркал,
как тишины пустой вокзал,
тебя там долго кто-то ждал,
так долго ждал – без остановок.

distortion only occurs when there is an attachment

[mflorell]

When I say "raw email text" I mean all of the formatting and header information that you don't see in an email viewer. If you use Gmail for example you can select "show original" and it will show you all of that information.

[mister511]

Thanks to all

Code: Select all

#!/usr/bin/perl
use DBI;
use Mail::IMAPClient;
use MIME::Base64;
use Encode qw(from_to decode encode);
use Email::MIME;
use Email::MIME::ContentType;
use Email::Address;
use HTML::Strip;
use DateTime::Format::Mail;
#use Data::Dumper;

binmode(STDOUT, ":utf8");

sub StripHTML {
  my ($text) = @_;
   $text =~ s/^\s*(.*?)\s*$/$1/;
   $text =~ s/[\r\n]/ /g;
   $text =~ s/\&nbsp\;/ /gi;
   if ($text =~ /<html.*?<\/html>/i) {
      my $hs = HTML::Strip->new;
      $text = $hs->parse($text);
      $hs->eof;
   }
  return $text;
}

$PATHconf = '/etc/astguiclient.conf';
$force_check = 1;
$ssl_no_cert = 0;
$DBX = 0;

open(conf, "$PATHconf") || die "Can't open $PATHconf: $!\n";
@conf = <conf>;
close(conf);

$i=0;
foreach(@conf) {
  $line = $conf[$i];
  $line =~ s/ |>|\n|\r|\t|\#.*|;.*//gi;
  if (($line =~ /^PATHhome/) && ($CLIhome < 1)) {
    $PATHhome = $line;
    $PATHhome =~ s/.*=//gi;
  }
  if (($line =~ /^PATHlogs/) && ($CLIlogs < 1)) {
    $PATHlogs = $line;
    $PATHlogs =~ s/.*=//gi;
  }
  if (($line =~ /^PATHagi/) && ($CLIagi < 1)) {
    $PATHagi = $line;
    $PATHagi =~ s/.*=//gi;
  }
  if (($line =~ /^PATHweb/) && ($CLIweb < 1)) {
    $PATHweb = $line;
    $PATHweb =~ s/.*=//gi;
  }
  if (($line =~ /^PATHsounds/) && ($CLIsounds < 1)) {
    $PATHsounds = $line;
    $PATHsounds =~ s/.*=//gi;
  }
  if (($line =~ /^PATHmonitor/) && ($CLImonitor < 1)) {
    $PATHmonitor = $line;
    $PATHmonitor =~ s/.*=//gi;
  }
  if (($line =~ /^VARserver_ip/) && ($CLIserver_ip < 1)) {
    $VARserver_ip = $line;
    $VARserver_ip =~ s/.*=//gi;
  }
  if (($line =~ /^VARDB_server/) && ($CLIDB_server < 1)) {
    $VARDB_server = $line;
    $VARDB_server =~ s/.*=//gi;
  }
  if (($line =~ /^VARDB_database/) && ($CLIDB_database < 1)) {
    $VARDB_database = $line;
    $VARDB_database =~ s/.*=//gi;
  }
  if (($line =~ /^VARDB_user/) && ($CLIDB_user < 1)) {
    $VARDB_user = $line;
    $VARDB_user =~ s/.*=//gi;
  }
  if (($line =~ /^VARDB_pass/) && ($CLIDB_pass < 1)) {
    $VARDB_pass = $line;
    $VARDB_pass =~ s/.*=//gi;
  }
  if (($line =~ /^VARDB_port/) && ($CLIDB_port < 1)) {
    $VARDB_port = $line;
    $VARDB_port =~ s/.*=//gi;
  }
  $i++;
}

if (!$VARDB_port) {
  $VARDB_port = "3306";
}

$dbh = DBI->connect("DBI:mysql:$VARDB_database:$VARDB_server:$VARDB_port", "$VARDB_user", "$VARDB_pass", { mysql_enable_utf8 => 1 }) or
  die "Couldn't connect to database: " . DBI->errstr;

# Get the settings from system_settings
$stmt = "SELECT allow_emails, default_phone_code, enable_queuemetrics_logging, queuemetrics_server_ip, queuemetrics_dbname, queuemetrics_login, queuemetrics_pass, queuemetrics_log_id, queuemetrics_eq_prepend FROM system_settings;";
$sth = $dbh->prepare($stmt) or die "Can't prepare: ", $dbh->errstr;
$sth->execute or die "Can't execute: $stmt", $dbh->errstr;
$sthArows=$sth->rows;
if ($sthArows > 0) {
   @aryA = $sth->fetchrow_array;
   $SSallow_emails = $ary[0];
   $SSdefault_phone_code = $ary[1];
   $enable_queuemetrics_logging = $ary[2];
   $queuemetrics_server_ip   = $ary[3];
   $queuemetrics_dbname = $ary[4];
   $queuemetrics_login= $ary[5];
   $queuemetrics_pass = $ary[6];
   $queuemetrics_log_id = $ary[7];
   $queuemetrics_eq_prepend = $ary[8];
}
$sth->finish();

# Exit program if system setting for allow email is not enabled
if ($SSallow_emails < 1 && $force_check < 1) {
  exit;
}

$stmt = "SELECT email_account_id, email_account_name, email_account_description, user_group, protocol, email_replyto_address, email_account_server, email_account_user, email_account_pass, pop3_auth_mode, active, email_frequency_check_mins, group_id, default_list_id, call_handle_method, agent_search_method, campaign_id,list_id,email_account_type from vicidial_email_accounts where active = 'Y' and protocol = 'IMAP'";
$sth = $dbh->prepare($stmt);
$sth->execute();

while (@row = $sth->fetchrow_array) {
   $VARemail_ID = $row[0];
   $VARemail_protocol = $row[4];
   $VARemail_server = $row[6];
   $VARemail_user = $row[7];
   $VARemail_pwd = $row[8];
   $VARpop3_auth_mode = $row[9];
   $VARemail_frequency = $row[11];
   $VARemail_groupid = $row[12];
   $default_list_id = $row[13];
   $call_handle_method = $row[14];
   $agent_search_method = $row[15];
   $campaign_id = $row[16];
   $list_id = $row[17];

  %args = (
    Server => "$VARemail_server",
    User => "$VARemail_user",
    Password => "$VARemail_pwd",
    Port => 993,
    Ssl => 1,
    Ignoresizeerrors => 1,
    Debug => "$DBX",
  );

  my $imap = Mail::IMAPClient->new(%args) or
    die "Cannot connect through IMAPClient: ($VARemail_ID | $VARemail_server | $VARemail_user | ssl-no-cert:$ssl_no_cert) $@ $!";

  if ($imap->IsAuthenticated()) {
    my @inbox_folders = $imap->folders("INBOX");
    for (my $i = 0; $i < scalar(@inbox_folders); $i++) {
      $imap->select($inbox_folders[$i]);

      my @msg_ids = $imap->search("(UNSEEN)");
      foreach my $msg_id(@msg_ids) {
        my $attach_ct = 0;
        my @ins_values = ();
        my $parser = Email::MIME->new($imap->message_string($msg_id));

        # Parse From
        my @addrs = Email::Address->parse($parser->header_str('From'));
        my $email_from = $addrs[0]->address;
        my $email_from_name = $addrs[0]->name;

        my $email_to = $parser->header_str('To');
        my $email_subject = $parser->header_str('Subject');
        my $email_date =  DateTime::Format::Mail->parse_datetime($parser->header_str('Date'))->strftime('%Y-%m-%d %H:%M:%S');
        my $email_mime = $parser->header_str('MIME-Version');
        my $email_content_type = $parser->header_str('Content-Type');
        my $email_cte = $parser->header_str('Content-Transfer-Encoding');
        my $email_x_mailer = $parser->header_str('X-Mailer');

        # Parse IP address
        my $email_sender_ip = '';
        my ($ip) = $parser->header_str('Authentication-Results') =~ /(([1-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9])){3})/i;
        if (defined($ip)) {
          $email_sender_ip = $ip;
        } else {
          my ($ip) = $parser->header_str('Received-SPF') =~ /(([1-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9][0-9]|[0-9])){3})/i;
          if (defined($ip)) {
            $email_sender_ip = $ip;
          }
        }

        print "email_from = " . $email_from . "\n";
        print "email_from_name = " . $email_from_name . "\n";
        print "email_to = " . $email_to . "\n";
        print "email_date = " . $email_date . "\n";
        print "email_mime = " . $email_mime . "\n";
        print "email_content_type = " . $email_content_type . "\n";
        print "email_x_mailer = " . $email_x_mailer . "\n";
        print "email_sender_ip = " . $email_sender_ip . "\n";

        my $email_message = '';

        $parser->walk_parts(sub {
          my ($part) = @_;
          return if $part->subparts;

          my $attachment_type = '';
          my $attachment_fulltype = '';
          my $attachment_filename = '';
          my $attachment_filesize = '';
          my $attachment_cte = '';
          my $attachment_contents = '';

          # Prefer text/html
          if ($part->content_type =~ m[text/html]i) {
            $email_cte = $part->header_str('Content-Transfer-Encoding');
            $email_message = StripHTML($part->body_str);
          } elsif ($part->content_type =~ m[text/plain]i && $email_message eq '') {
            $email_cte = $part->header_str('Content-Transfer-Encoding');
            $email_message = $part->body_str;
          } else {
            my ($ctype) = $part->content_type =~ m{(application/(pdf|msword|rtf|vnd.ms-powerpoint|vnd.ms-excel|x-msexcel|ms-excel|zip|(vnd.oasis.opendocument.(text|spreadsheet)))|image/(jpeg|jpg|gif|x-png|png|bmp|x-ms-bmp)|text/(csv|plain))}i;
            if (defined($ctype)) {
              $attachment_type = $ctype;
              $attachment_filename = $part->filename;
              $attachment_cte = $part->header_str('Content-Transfer-Encoding');
              $attachment_contents = $part->body_raw;

              my $dis = $part->header("Content-Disposition") || '';
              my $attrs = parse_content_disposition($dis)->{attributes};
              #print Dumper($attrs);
              $attachment_filesize = $attrs->{size} || '';

              if($attachment_type =~ m{application/pdf}i) {
                $attachment_fulltype = "PDF";
              } elsif ($attachment_type =~ m{image/(jpeg|jpg)}i) {
                $attachment_fulltype = "JPG";
              } elsif ($attachment_type =~ m{image/gif}i) {
                $attachment_fulltype = "GIF";
              } elsif ($attachment_type =~ m{image/(x-png|png)}i) {
                $attachment_fulltype = "PNG";
              } elsif ($attachment_type =~ m{image/(bmp|x-ms-bmp)}i) {
                $attachment_fulltype = "BMP";
              } elsif ($attachment_type =~ m{application/msword}i) {
                $attachment_fulltype = "DOC";
              } elsif ($attachment_type =~ m{application/rtf}i) {
                $attachment_fulltype = "RTF";
              } elsif ($attachment_type =~ m{application/vnd.ms-powerpoint}i) {
                $attachment_fulltype = "PPT";
              } elsif ($attachment_type =~ m{application/(vnd.ms-excel|x-msexcel|ms-excel)}i) {
                $attachment_fulltype = "XLS";
              } elsif ($attachment_type =~ m{application/zip}i) {
                $attachment_fulltype = "ZIP";
              } elsif ($attachment_type =~ m{text/csv}i) {
                $attachment_fulltype = "CSV";
              } elsif ($attachment_type =~ m{text/plain}i) {
                $attachment_fulltype = "TXT";
              } elsif ($attachment_type =~ m{application/vnd.oasis.opendocument.text}i) {
                $attachment_fulltype = "ODT";
              } elsif ($attachment_type =~ m{application/vnd.oasis.opendocument.spreadsheet}i) {
                $attachment_fulltype = "ODS";
              }

              if ($attachment_type ne '' && $attachment_filename ne '') {
                $ins_values[$attach_ct] = "'$attachment_filename','$attachment_type','$attachment_cte','$attachment_filesize','$attachment_fulltype','$attachment_contents'";
                $attach_ct++;
              }
            }
          }

          print "content_type = " . $part->content_type . "\n";
          if ($attachment_type ne '') {
            print "attachment_type = " . $attachment_type . "\n";
            print "attachment_fulltype = " . $attachment_fulltype . "\n";
            print "attachment_filename = " . $attachment_filename . "\n";
            print "attachment_filesize = " . $attachment_filesize . "\n";
            print "attachment_cte = " . $attachment_cte . "\n";
            print "substr(attachment_contents, 0, 50) = " . substr($attachment_contents, 0, 50) . "\n";
          }
        });

        $email_message =~ s/^\s+|\s+$//g; # Remove leading and trailing whitespaces
        print "email_cte = " . $email_cte . "\n";
        print $email_message . "\n";

        #open($file, ">", "./_mail/" . $msg_id . "_raw");
        #$imap->message_to_file($file, $msg_id);
        #close($file);

        ######## Save to DB ########
        my $status = "NEW";
        my $limit = 1;
        $call_handle_clause = "";
        if ($call_handle_method eq "EMAILLOOKUP") {
          $call_handle_clause = "where email = '$email_from' order by lead_id desc";
        } elsif ($call_handle_method eq "EMAILLOOKUPRL") {
          $call_handle_clause = "where email='$email_from' and list_id = '$list_id' order by lead_id desc";
        } elsif ($call_handle_method eq "EMAILLOOKUPRC") {
          my $list_id_stmt = "select list_id from vicidial_lists where campaign_id = '$campaign_id'";
          my $list_id_rslt = $dbh->prepare($list_id_stmt);
          $list_id_rslt->execute();
          $list_id_str = "";
          while (@list_id_row = $list_id_rslt->fetchrow_array) {
            $list_id_str .= "$list_id_row[0],";
          }
          $list_id_str = substr($list_id_str, 0, -1);
          $call_handle_clause = "where email = '$email_from' and list_id in ($list_id_str) order by lead_id desc";
        } else {
          $limit = 0;
        }

        # Check if lead exists in vicidial_list table via a search by email based on the email account settings
        my $vicidial_lead_check_stmt = "select lead_id, list_id from vicidial_list $call_handle_clause limit $limit";
        my $vicidial_lead_check_rslt = $dbh->prepare($vicidial_lead_check_stmt);
        $vicidial_lead_check_rslt->execute();
        if ($vicidial_lead_check_rslt->rows > 0) {
          my @lead_id_row = $vicidial_lead_check_rslt->fetchrow_array;
          $lead_id = $lead_id_row[0];
        } else {
          my $vicidial_list_stmt = "insert into vicidial_list(list_id, email, comments, status, entry_date, phone_code) values('$default_list_id', '$email_from', '".substr($message,0,255)."', '$status', NOW(), '$SSdefault_phone_code')";
          my $vicidial_list_rslt = $dbh->prepare($vicidial_list_stmt);
          if ($vicidial_list_rslt->execute()) {
            $lead_id = $dbh->last_insert_id(undef, undef, 'vicidial_list', 'lead_id');
          } else {
            die "Vicidial list insert failed. Check SQL in:\n $vicidial_list_stmt\n";
          }
        }

        # Insert a new record into vicidial_email_list. This is ALWAYS done for new email messages.
        $ins_stmt = "insert into vicidial_email_list(lead_id, protocol, email_date, email_to, email_from, email_from_name, subject, mime_type, content_type, content_transfer_encoding, x_mailer, sender_ip, message, email_account_id, group_id, status, direction) values('$lead_id', 'IMAP', STR_TO_DATE('$email_date', '%Y-%m-%d %H:%i:%S'), '$email_to', '$email_from', '$email_from_name', '$email_subject', '$email_mime', '$email_content_type', '$email_cte', '$email_x_mailer', '$email_sender_ip', trim('$email_message'), '$VARemail_ID', '$VARemail_groupid', '$status', 'INBOUND')";
        my $ins_rslt = $dbh->prepare($ins_stmt);
        if ($ins_rslt->execute()) {
          $email_id = $dbh->last_insert_id(undef, undef, 'vicidial_email_list', 'email_row_id');
          if ($attach_ct > 0) {
            $multistmt = "";
            for ($k=0; $k < $attach_ct; $k++) {
              $ins_values[$k] = "('$email_id', $ins_values[$k])";
              $multistmt .= "$ins_values[$k],";
              $output_ins_values[$k] = "('$email_id', $output_ins_values[$k])";
              $output_multistmt .= "$output_ins_values[$k],";
            }
            $attachment_ins_stmt = "insert into inbound_email_attachments(email_row_id, filename, file_type, file_encoding, file_size, file_extension, file_contents) VALUES ".substr($multistmt,0,-1);
            $attachment_ins_rslt = $dbh->prepare($attachment_ins_stmt);
            $attachment_ins_rslt->execute();
          }
        }
      }
    }
  }
  $imap->logout();
}


[carpenox]

what does that script do mister?

[mister511]

what does that script do mister?

/usr/share/astguiclient/AST_inbound_email_parser.pl
Instead of this script.
Displays text with attachments normally