vicidial.org

by **aouyar** » Fri Jul 03, 2009 3:28 am

I suspect that the current translation scheme might cause some Consistency Issues or Bugs in Multi-language installations.

Reviewing the translated PHP files in 2.0.5.1rc1 and double-checking the information stored in the database it seems that differing user status/event information might end up in the database depending on the interface language of the agent. This might create a big mess if some of the agents log in using one language and other agents use another language. Even more complicated situations might arise if the same agent logs in using another language from time to time.

This might create a big mess for the Call Center Supervisor when generating reports on users. It might even cause some hard to detect bugs in certain processes because it seems like in some cases of the status/event values are used in branching in application logic.

For consistency the state and event information stored in the database must be independent of the language of the user interface, but the information must be presented to the user in his respective language in runtime.

The specific case I have encountered is in the Spanish translation of the agent interface:
- When the agent is logging out different status strings (LOGOUT in English and SALIR in Spanish) get registered in the following tables:
vicidial_user_log (event column)
vicidial_timeclock_status (status column)
vicidial_timeclock_log (event column)
vicidial_timeclock_audit_log (event column)
- The same problem applies to the START events. They end up being registered as COMIENZO through the Spanish Interface.
- The LOGIN events do not present the problem because the mapping in Spanish is not in the translation file.

Has anyone else detected the same problem or could anyone confirm the issue? I checked the Mantis issue tracker, but I have not encountered any related issues. I am writing to the development list, because - considering for a moment that the issues is confirmed - many similar bugs might end up in the code as a side effect of the translation method that is being used. The people involved in the translations usually do not dig in to the code and any error that they might introduce in the translation files might produce similar bugs due to the search-and-replace translation build process.

"Has using gettext for translations been considered by the developers?" I know that this question was asked at least one more time in the list, but as far as I know it has not been responded. The gettext wrapper gives more control to the developer by explicitly enclosing the texts that are allowed to be translated in run-time.

I would be more than willing to hand a helping hand if the issue is confirmed.

by **mflorell** » Fri Jul 03, 2009 7:21 am

I am sorry that I do not mention that the rc releases do not have the code updated unless it is specifically mentioned in the documentation.

You can run your own translations however with the build_translation_www_files.pl script.

by **aouyar** » Sun Jul 05, 2009 3:25 pm

The issue I am describing is not tied to a specific version of Vicidial. I am perceiving a general issue in the language translation scheme.

It seems like the build_translation_www_files.pl might search and replace strings inserted to the database causing inconsistencies in the database. Specific examples of the issue have been cited in my previous post.

I might simply remove some entries to the translation files and run build_translation_www_files.pl to solve the specific cases listed in my previous post, but many similar bugs might be lurking in the code and new ones can be introduced by harmless looking changes in the translation files.

I would be glad to receive some opinions on the issue to start working towards a solution.

by **mflorell** » Mon Jul 06, 2009 2:58 pm

You are correct, there are some issues with how the translation files are processed, perhaps a blacklist in the build script itself might be a better solution, what do you think?

Do you have any other examples of words or phrases that should not be translated like this?

by **aouyar** » Mon Jul 06, 2009 9:10 pm

Hi Matt,

It is quite difficult to identify every case, where bugs might be introduced by the translation system. In fact this is the biggest maintainability issue with the current approach; each localization file might contain various bugs and more bugs can be introduced by inadvertent modifications in the translation files.

In my opinion we have to lower the bar for translators. With the current approach, the localization process is error prone and it is not possible to develop a new translation file without digging into the code to check potential conflicts.

The blacklist approach might help, but then the build_translation_www_files.pl is already somewhat complex and adding blacklists could make it more difficult to maintain. Furthermore no blacklist will deal with all the corner cases, and constructing blacklists to handle most of the cases could be a difficult task.

In my humble opinion the only way to guarantee that simple changes in localization files will not introduce logic errors in the application is using a Whitelist Approach. The translation mechanism must not be allowed to translate any text that is not explicitly marked to be translated by the code developer. The gettext library achieves this by encapsulating any translatable text within gettext( .... ).

In my opinion the most standards based approach to solving the problem would be using the gettext library. I am sure converting all the code base to using gettext would require less effort and time than constructing blacklist to handle an acceptable proportion of the corner cases.

Have you considered using the gettext library in Vicidial project? (Apparently the same question was asked once in the forum a few years ago, but no one had replied.)

If you have any objections to using gettext or any other solution that works in run-time, another option might be explicitly marking texts that require translation in the code. I was thinking on a unobtrusive scheme that permits marking text and generating translated php files offline, but I would really like to explore the gettext route first.

My company is in the beginning stages of a Call Center implementation in Colombia and we are willing to overtake the overhaul of the translation scheme of the agent interface.

Have a nice day

by **mflorell** » Tue Jul 07, 2009 6:24 am

Well, there are several reasons I went with a static build as opposed to gettext. Mostly the fact that there are 3000+ phrases in the admin and agent interfaces that would add significant load to the system to quickly process and reload in each PHP script as well as the significant amount of development time that it would take to rewrite all translations to use the gettext method.

Is this something that you would be able to put time into to see if it is a viable option?

by **aouyar** » Tue Jul 07, 2009 9:13 am

Of the 3000+ phrases 4 in 5 are in the admin interface, so the impact of using gettext might be lower than one might initially assume.

Once all the translatable text are encapsulated with gettext() or the shorthand notation _(), the script build_translation_www_files.pl can be modified to replace all invocations for gettext with static text eliminating the runtime overhead completely.

This approach has the following benefits:
- More maintainable code and translations. The application logic would be isolated from translation bugs.
- The barrier for translator would be lowered. Contributors can work on localizations using standard tools they are accustomed to use.
- Without any translation files in place, the code will work as is for testing in English with minimal overhead.
- With language translation files in place, the code will work localized at run-time with a slight runtime overhead.
- In case zero run-time overhead is required one can use the new build_translation_www_files.pl to generate static translated files for production.

The company I am working in has committed to a project in which the client requires that the agent interface be in Spanish. Localized agent interface is one of the basic requirements of many potential clients; localized administrator interface is of lower priority for us at the moment. I would rather invest time into converting the code to use gettext than doing an overall code audit trying to identify all the corner cases in translations that might result in bugs.

Once we agree on the approach the biggest issue would be deciding which version of astguiclient to work on and on this point I need your advice. I would be inclined to use the stable 2.0.5 version at the client site, but for adoption by the Vicidial project it would be better to work on the trunk version.

I used the 2.0.5 series at a small client site when it was in the last phases of development with very good results, but I would like to avoid risks due to using the development version for the new project. On the other hand I am really impressed by the pace of development and the things you have been doing in the development branch. Could you tell me what the actual level of stability of the 2.2 branch is? When is the expected release date of this branch?

by **mflorell** » Tue Jul 07, 2009 10:53 am

the 2.2.0 trunk branch is very stable and is in use at over 20 client sites right now. We typically do not put code into SVN trunk if it is not tested in production for at least a day, and if there are issues in production then they are fixed very quickly.

If you are going to proceed with this I would strongly recommend using SVN trunk to base your work on.

by **aouyar** » Sun Oct 11, 2009 8:59 pm

Hi Matt,

I will be posting the internationalised version of the Agent Interface soon. I skimmed through most of the code and replaced al references to texts displayed to client by calls to the gettext macro.

The difficult part of the work was going through all the source and throwing in the gettext commands without modifying the functioning or code style of the existing code.

I used the poedit program to parse the source code and generate the messages.po and messages.mo files for es_ES.utf8 locale. I will be finishing the translation of all the strings to Spanish soon. With this method the agent interface can be translated into any other language safely by a non-programmer in one night.

The Spanish translations may have some errors and I am sure there is room for plenty of improvement. Even though I had the opportunity to consult some native speakers, I am not a native Spanish Speaker myself, so anyone that is willing to contribute to improving the translations is more than welcome to do so.

I hope the code will be adopted by the project, because:
In my opinion it might be a great step towards fully internationalised versions of vicidial in many languages.
* There should not be any noticeable impact on performance. You can check the following article on Bencmarking PHP Localization for reference: http://mel.melaxis.com/devblog/2006/04/ ... st-enough/
* Maintaining a separate internationalised version of the code can be a real headache.

I will installing the internationalised version of the agent interface at a client site in Colombia on tuesday, so by then I would definitely have some material to post on Mantis. I would be glad if you could check the code as soon as possible and indicate how we can proceed.

I am willing to offer a helping hand to anyone that would like to contribute to the internationalisation effort.

Bye

by **mflorell** » Mon Oct 12, 2009 5:48 am

Thanks for the update!

by **kimhoogenberg** » Mon Oct 12, 2009 3:23 pm

Seems like a very good initiative to me! I've also struggled with the way translation was done. Matt, did you get my better Dutch translation from Demian by the way? I'm kind of like a Dutch language purist, so I must be better now and going into production in a few weeks. Also better now, because it seemed some of the previous Dutch translations looked as if they were done by Google..

Futhermore, I am a bit against putting all full sentences in between gettext brackets or any other method. Wouldn't it be better to use some kind of functional name/value pairs? Like "agc.label.recording" instead of just "RECORDING FILE:"? One needs to translate agc.label.recording to RECORDING FILE: for English too, but this really lowers mistakes, makes code easier to read and maintain, because it is shown in its functional context.

When you read the generated PHP code, it seems a bit hardish at first not to see fully translated stuff, but going through the localized files, it is way easier that way. Also, PHP code is not invented to read like a book.

by **aouyar** » Mon Oct 12, 2009 4:39 pm

Hi kimhoogenberg,

What you are recommending seems to be a cleaner approach. But I would prefer to start off with embedding english texts in gettext for the following reasons:
* When you are dealing with hundreds or thousands of strings, coming up with distinct, descriptive names becomes complicated.
* Embedding english text directly works out of the box without a language file for English Language.
* Embedding the english text directly in the code work sensibly most of the time (and even when it does not, the work-around is quite simple, a minor modification in the english text does wonders).
* The code would not be easily recognizable/readable for other developers if I go on and strip all the English language strings and merging code in the transition period might become more difficult.
* Once the code is fully internationalised it is quite easy to generate a language file for english automatically and implement improvements over the current scheme. (The automatically generated language files contain line numbers for all references to gettext.)
* I would recommend to use the english text as the label as a general rule and only for the special cases where there are context problems I would recommend to use the more concise context+string as the label.

Matt has the final word on the approach he would be more comfortable with and the code he will be willing to merge, but in my opinion replacing all texts with labels would make the maintenance of the code more complicated.

Bye

by **kimhoogenberg** » Mon Oct 12, 2009 4:46 pm

Hi!

Good topic though

Imagine that gettext() would go by the name of "GeT me SOMe Text PlEAse()". How often would one spell that incorrect (assuming no IDE is used)?

Same conclusion I can only make towards i18n. It is going to frustrate more and more using this way. Though, for the first steps, I would indeed just replace the method used by surrounding with gettext(), just for speed.

Regards,

Kim

by **aouyar** » Mon Oct 12, 2009 5:49 pm

I agree that it could be great to get long message texts out of the code and into the language files at some point.

by **mflorell** » Mon Oct 12, 2009 5:52 pm

Yes, I did get the new Dutch translation, thanks

Haven't had a chance to do anything with it yet, been very busy going to conferences and traveling to clients.

As for the gettext() method, for the agent interface it is probably the best choice in the end. There are only a few hundred phrases and they are not that big. However, for the admin interface there are thousands of phrases, and many of them are quite long which makes this option a bit more difficult to implement.

It might be best to proceed with something like the gettext method for the agent and stay with the current method for admin until we can figure out a better option. Plus it will take A LOT of manual hours to do the Admin gettext initial modification.

by **kimhoogenberg** » Mon Oct 12, 2009 7:24 pm

Matt, I agree and totally ready to help..

by **williamconley** » Mon Oct 12, 2009 11:53 pm

have we looked at some other serious software that has multiple languages available? there are versions that use sql to store the languages, but there are some closer to home implementations such as SugarCRM that have a nice php'ish one very close to what you are talking about. not a difficult one to implement really, as it COULD be done incrementally

would be awkward during the transition, but only because it's not done ... each time more of the text is removed from the page and placed in the translator file, it just gets easier to manage that file.

in sugar, each "module" has a "language" folder. i guess we won't need that, but ...

each language folder has a file specific to the file from whom they have removed the text. but each is now preceded by "en_us."

They even have a ".help." for most of the files, which is another interesting concept. Each file contains simply an array of strings with a standard struct:

'LBL_CHECKOUT_DATE'=>'Checkout Date',

this is something that could be implemented one file at a time. perhaps even one section at a time. future versions would then be able to pop this into a table instead if someone wanted to get really silly. in the meantime, a new language would just be copying the file to "en_uk" and then translating the text (with the screen visible in case you don't really know the context for the translation).

i guess the only serious implementation concept would be to carry a language variable through all screens so we don't lose it, and then just start translating files. as each file comes into the fold, add code to pull the english language file as an include unless there is a language code active, and then verify the language file exists or use english anyway.

any file that hasn't been translated yet just gets replaced by a manually translated version until it's all done.

i can't do translation, but i could wander around and "strip" some peripheral files for english. i will need to get a 2.2.X svn box online to do it, though, i guess.

is there already a version of this underway?

by **aouyar** » Tue Oct 13, 2009 12:46 am

No need to re-invent the wheel. Gettext works good; it is much more efficient than storing string mappings in arrays, because it implements caching; there are many existing command line and GUI tools to assist the translation process, and there is an existing internationalised version of the agent interface based on gettext that I would be publishing within 48 hours.

by **aouyar** » Tue Oct 13, 2009 2:07 am

Hi Matt,

The initial version of the internationalisation code is posted to Mantis:
http://www.vicidial.org/VICIDIALmantis/view.php?id=240

The following files for agent interface were modified for internationalisation:
vicidial.php
vdc_db_query.php
timeclock.php
conf_exten_check.php

I also modified the welcome.php file in the vicidial folder.

I did not update astguiclient.php, because I understand that it is not used so frequently now.

I did not touch the debugging and warning messages, only the messages that are displayed to user are encapsulated by gettext. As a general rule all messages exchanged internally between scripts or written to database were left unchanged.

The localization of the interface is configured in the new localization.php file included by the internationalised files. For the moment the default locale is switched by modifying the locale variable.

I would recommend to have the default locale and the available locales stored in the system or server settings table of the database, but this is not implemented yet. I have not implemented run-time switching of interface language from the web interface either, but it would be quite easy to implement as a drop-down in the login screen. I would be waiting for suggestions on these points, before going ahead with the implementation.

I did not change the logic of the code, except for one instance where I have identified a bug. Line 9336 of vicidial.php which was:
if (eregi("X",dial_prefix))
was changed as:
if (eregi("X",dial_prefix))
(The notification message must not be shown when the dialing prefix is disabled.)

The code is based on the trunk version (2.2 rev 1240). I could update the code to match rev 1244 that was published recently or a later version after receiving some feedback.

Matt, would you consider to commit the internationalization changes to trunk? Do you have any suggestions on how I can proceed?

by **mflorell** » Wed Oct 14, 2009 9:02 pm

I'm at Astricon this week so I won't have much time to take a look at it until next week. If I don't comment by next Wednesday please ping this topic.

by **aouyar** » Wed Oct 14, 2009 11:27 pm

Hi Matt,

I will be looking forward to feedback and suggestions from you next week.

Hope you enjoy Astricon,
Ali

by **aouyar** » Wed Oct 28, 2009 3:39 pm

Hi Matt,

Any updates on this one?

Bye

by **williamconley** » Wed Oct 28, 2009 6:26 pm

let's see ... one week at astricon ... i'd estimate at least three weeks trying to "catch up". probably closer to four.

by **aouyar** » Wed Oct 28, 2009 6:48 pm

Good to know about it. It is always better to have more realistic expectations. ;-)

by **mflorell** » Wed Oct 28, 2009 10:55 pm

I actually just received a new chinese translation and another updated translation as well. On top of some new client work and a project aht never seems to end I am a bit busy. It is on my list though

by **aouyar** » Thu Oct 29, 2009 6:56 am

Thanks for the update. You can write to me if there is anything I can give a hand with; ex. an updated version of the patch for the last trunk revision, code to store the default locale in the database, clarifications about the code, etc.

Implementing internationalised using gettext, you can free yourself from the burden of maintaining translations to a great deal. It could be easier on you and easier on the translators.

I am willing to contribute code on other areas like bugfixes, improvements in dialplan generation and audio store, but doing out of tree development and maintaining code against a moving target (SVN Trunk) is complicated.

by **aouyar** » Mon Nov 30, 2009 3:38 pm

Hi Matt,

I am planning to update the internationalization code to catch up with Trunk. My initial work was based on Trunk r1240.

Are you expecting any significant changes in SVN soon? Is it a good moment to resync or shall I wait for any imminent change?

Have you had an opportunity to check the code? Do you think the code could be included in the 2.2 release?

TIA

by **mflorell** » Mon Nov 30, 2009 3:52 pm

I am actually planning on some significant changes in SVN trunk. There are a few big projects that should be done in the next couple weeks that will result in some bigger changes in the ViciDial Agent screen.

by **aouyar** » Mon Nov 30, 2009 3:57 pm

That brings us to the other question: Is it feasible to merge in the Internationalization Code before the big changes that you are planning?

by **mflorell** » Mon Nov 30, 2009 4:59 pm

Unfortunately I don't have the time to do that and the projects that I have to finish in the next two weeks. But we will halt development in a couple weeks so we can get 2.2.0 release out, and at that time we should be able to integrate your internationalization code.

by **aouyar** » Mon Nov 30, 2009 5:05 pm

Sounds great.

by **Michael_N** » Thu Dec 03, 2009 4:37 pm

mflorell wrote:I am actually planning on some significant changes in SVN trunk. There are a few big projects that should be done in the next couple weeks that will result in some bigger changes in the ViciDial Agent screen.

What kind of changes is that?

by **mflorell** » Thu Dec 03, 2009 6:02 pm

Nothing on the visual side really, actually more on the back-end of the agent interface. There will only be a few actual changes in the vicidial.php script itself.

The first change is the addition of an API method(and eventually a manager screen) with the ability to change agent selected in-groups in real-time(but not the ability to change the campaign). There is a lot of code involved in making this change, which is why I wanted to mention it.

The next one is the ability to tie-in to dial through an outside API(this is for a custom client application, but will involve some new features to the agent interface as well)

by **aouyar** » Fri Jan 08, 2010 9:37 am

Hi Matt,

Do you think it is possible to integrate the gettext based internationalization code in the 2.2 release? It is quite difficult to track SVN and maintain the internationalization code out of tree.

I could make a patch against the last revisión of 2.2 this weekend.

by **mflorell** » Fri Jan 08, 2010 10:42 am

If you can get me a patch this weekend I'll try to get it in, but the translation actually went pretty easily this time. I even added Traditional Chinese without any problems

by **aouyar** » Fri Jan 08, 2010 11:12 am

The translation scheme just works, I have no doubt about it, but as I have mentioned in my previous posts, the search-and-replace method employed by the scripts inadvertently change strings that are written to database causing inconsistencies in the database.

And finally it is about lowering the bar for maintenance of translations. Once gettext is implemented non-programmers can develop new translations or update existing translations easily.

I will work hard to get the patch ready for monday to get it in. Some additional changes will be required to store the default language in the system settings.

Thanks in advance

by **aouyar** » Tue Jan 12, 2010 8:36 am

Hi Matt,

I am still working on updating the gettext patch for 2.2rc2. The changes in vicidial.php were much greater than I expected, so I have to double check many code segments manually. I hope to have it ready before tomorrow.

Have a nice day

by **aouyar** » Tue Jan 12, 2010 10:11 pm

I uploaded a new version of the code for 2.2rc2 (rev. 1302) to Mantis:
http://www.vicidial.org/VICIDIALmantis/view.php?id=240

In addition to the agent interface, welcome.php was internationalized. The following pieces are missing, but I can give a hand with the development as soon as we agree on the language configuration scheme:
* Change in database schema for storing the default locale for system.
* Change in Admin Interface for setting default locale.
* Modifications for setting language in the login screen.

Presently the locale is hard-coded in localization.php file. The locale configuration code can be included by the other PHP files.

The code has only received basic testing, but I don't think any serious bugs are lurking in the code, because no changes to application logic was mode; only the message strings where encapsulated in gettext.

You can test the code by over-writing the WWW files with the files in the attached agc_intl_r1302.tgz file.

There so many changes in the agent interface since rev. 1240 that I was having some difficulties in merging the code. That is until I switched to using kdiff3 for doing a 3-way merge.

I am using poedit for creating and editing the language files. Creating a new localization is as easy as creating a new po file, pointing it to the source directory to generate a list of the strings that must be translated directly from the source code. This tool is quite intuitive for modifyinh and updating the translation strings.

I will be waiting for feedback. If I can do anything else to get the code in 2.2, just let me know.

by **mflorell** » Wed Jan 13, 2010 4:15 am

I would really rather wait on this piece for 2.4 if that is a possibility. i just don't have the time at this point to add this in unless we delay 2.2.0 for another couple weeks.

The plan right now is to have 2.4 out in the summer.

by **aouyar** » Wed Jan 13, 2010 10:12 am

Hi Matt,

This is kind of sad news. :-(

I would really have liked to have this integrated into 2.2. I was thinking of finding a way for generating the translation files for gettext automatically from the existing xx_language.txt translation files. If there are any changes in your release plans for 2.2 just let me know.

In fact there is a case for including the internationalisation code in 2.2: After the stable version is released the translations are rarely updated, but with gettext in place people could go on improving the localization or adding new localizations without touching the vicidial source.

In case we have to leave this for 2.4, I would recommend that the internationalisation code be merged in trunk right from the beginning, so that the code will evolve with the code base (instead of diverging from it) and the people could start contributing the translations early on. The 2.4 branch would probably start out with a much more rapid pace of development and I fear that if we do not merge the code right from the beginning, it could become totally irrelevant in a short time.

Bye

vicidial.org

Multi-language Site Consistency Issue

Multi-language Site Consistency Issue

Internationalised Version of Agent Interface Almost Complete

Who is online