Handling encoding issues
========================

LemonLDAP::NG has to integrate with many external systems: HTTP servers and
clients, databases, libraries (such as Lasso). At each integration point, it is
crucial to ensure that text data is provided in the right form: text string,
binary string, etc.

Passing an ISO-8859-1 string to a library that expects an UTF-8 string, or the
other way around, will lead to text encoding issues: text will contain odd,
unexpected or sometimes missing characters.

Text string and binary strings
------------------------------

Perl was invented before unicode was popular, therefore, even in modern systems,
Perl assumes that the default character set is ISO-8859-1. Perl is also
weak-typed, which means there is no clear difference between a binary string and
a text string. Therefore, when manipulating variables that contain text, the
developer must be aware of what format the text is supposed to be encoded as in
the current context.

**There is no way to distinguish whether a string contains text of binary data
in Perl**.

Because of this, we needs to have rules to decide if a given variable should
contain text data or binary data.

The Perl documentation recommends to immediately decode binary data to text
strings and manipulate text strings as much as we can.

However, external systems have no notion of text strings (or numbers, or
objects, or cats), computers only understand binary data. Therefore, everytime
you need to communicate with an external system, your text strings must be
encoded to, or decoded from, binary data.

Some libraries expect text strings, and some others expect binary data, sadly
the documentation is not always very clear about that. Some libraries let you
decide with an option.

For example, in Net::LDAP, the ``raw`` option can be used to indicate which
attributes contain binary data. The other attributs will return or expect text
strings. As another example, the ``pg_enable_utf8`` option makes DBI accept and
return text strings and not binary strings (except if the column type is a
binary type).

How to decode/encode data
-------------------------

In order to convert a text string to a binary string ::

    use Encode;

    # Session variables are stored as text strings
    my $title = $req->sessionInfo->{title};

    # Check that your original text string is correct
    $self->p->_dump_string($title);

    # Do not alter the original variable! Copy it and convert it to binary
    # before handing it to the library
    my $binary_title = encode("UTF-8", $title);

    function_of_my_lib($binary_title);

In order to convert a binary string to a text sting ::

    use Encode;

    my $result = function_of_my_lib();

    my $decoded_var = decode("UTF-8", $result);

    # Check that your decoded text string is correct
    $self->p->_dump_string($decoded_var);

    $req->sessionInfo->{title} = $decoded_var;


Checking how Perl understands a string
--------------------------------------

LemonLDAP::NG provides a ``_dump_string`` method that will help you figure out
what a variable contains. It requires the ``logLevel`` to be set to ``debug``.

::

    $self->p->_dump_string($var);
    [debug] $str = "Le\ Grand\ M\N{LATIN SMALL LETTER E WITH ACUTE}chant\ Loup"

You can use this method to verify if Perl has correctly understood the content
of your text strings. 

Note that is you try this method on a binary string, you will get erroneous
results. This is confusing, but expected.