Babak Mahmoudi’s Blog

Localization for Persian Language…

Archive for the ‘Persian Localization’ Category

Shift in .Net handling of culture data

leave a comment »

There’s been a shift of handling culture data in .Net 4. This may have influences on attempts in fixing Persian culture.

The shift is simple, while prior to version 4, CLR insisted on using privately stored culture data in a “culture.nlp” binary resource, version 4 gave up and uses windows API to get data from operating system. This is why some complained about missing CultureTableRecord while using PersianCalendar.

Culture Data Prior to .Net 4

Following sequence shows how culture data are handled in prior to version 4:

  1. CultureInfo constructor calls CultureTableRecord.GetCultureTableRecord

    image

  2. CultureTableRecord constructor uses CultureTable.Default.GetDataItemFromCultureName:image
  3. The Default property returns m_defaultInstance field which is initialized in class initializer:image
  4. Finally InitializeBaseInfoTablePointers loads “culture.nlp” from assembly resources:image
  5. where “culture.nlp” can be found in mscorlib resources:

image

There’s also a CalendarTable class with virtually same approach

image

 

CultureData in .Net 4

The above approach has been totally depreciated in .Net 4. Now CLR prefers to use windows API to retrieve culture data. For instance following excerpt is from disassembly of CalandarData code where CLR attempts to enumerate optional calendars by calling EnumCalendarInfoExEx:

image

Conclusion

There’s been a shift in handling culture data in .Net 4: Now CLR prefers to use windows API to get culture data This will have influences in fixing Persian culture as I will describe in later posts.

Written by Babak Mahmoudi

September 14, 2011 at 9:11 am

What’s wrong with Persian culture in .Net?

with one comment

This post was republished to Babak Mahmoudi’s Blog at 11:38:10 ق.ظ 08/22/2011

 

In this post, some mistakes in implementation of Persian culture in .Net are discussed and also get-around methods are proposed.

 

.Net provides enhanced globalization features mostly based on its implementation of Culture concepts. Programmers may use various aspects of these features to develop software ready for global market. A class called CultureInfo plays a key role in this implementation. It is mainly used to get necessary information about a specific culture. Programmers will create instances of CultureInfo, to access required information about a culture. For sure the framework supports the Persian language too. One may use ”fa-IR” to create a CultureInfo instance for Persian language in Iran. But at it is discussed here there are a number of problems with this culture instance.

The most critical deficiency of Persian culture is about Persian calendar. While Iranian people use their own calendar, Persian culture assumes they use Arabic Hijri calendar. Following picture shows how CultureInfo assumes HjriCalendar for Persian culture. Also note that PersianCalendar is not even included in OptionalCalendars.

clip_image002

Another problem with Persian culture is about calendar information such as day and month names. They all are Arabic ones:

clip_image004

clip_image006

So in order to have a better Persian CultureInfo one should:

· Find a way to set PersianCalendar for the culture calendar.

· Correct Months and Day names.

Correcting Months and Day names

Months and day names are actually included in DateTimeFormatInfo class property of CultureInfo. They can be easily fixed with code such as:

Culture.DateTimeFormatInfo.MonthNames = new string[] { "فروردین", "ارديبهشت", "خرداد", "تير", "مرداد", "شهریور", "مهر", "آبان", "آذر", "دی", "بهمن", "اسفند", "" };

Using Persian Calendar

Using Persian Calendar is not as straightforward as setting months names. Both CultureInfo and DateTimeFormatInfo include a calendar property. To get proper Persian date formatting one should set these calendars to Persian. One may assume to simply set the Calendar property :

Culture.DateTimeFormatInfo.Calendar = new PersianCalendar();

But the property set method of DateTimeFormatInfo prevents such settings because Persian Calendar is not included in OptionalCalendars of the Persian culture. One may use Reflection to by-pass the property set method to directly access the calendar property:

FieldInfo dateTimeFormatInfoCalendar = typeof(DateTimeFormatInfo).GetField("calendar",

BindingFlags.NonPublic | BindingFlags.Public | BindingFlags.Instance);

dateTimeFormatInfoCalendar.SetValue(info, new PersianCalendar());

Where info is a DateTimeFormatInfo. Note how reflection helps in setting a private firld “calendar” in a DateTimeFormatInfo object. This bypasses the set method logic of checking the OptionalCalendars.

Putting it altogether a candidate method for fixing the DateTimeFormatInfo can be:

Code Snippet
  1. public static void FixPersianDateTimeFormat(DateTimeFormatInfo info,bool UsePersianCalendar)
  2. {
  3.     FieldInfo dateTimeFormatInfoReadOnly = typeof(DateTimeFormatInfo).GetField("m_isReadOnly", BindingFlags.NonPublic | BindingFlags.Public | BindingFlags.Instance);
  4.     FieldInfo dateTimeFormatInfoCalendar = typeof(DateTimeFormatInfo).GetField("calendar", BindingFlags.NonPublic | BindingFlags.Public | BindingFlags.Instance); ;
  5.  
  6.     if (info == null)
  7.         return;
  8.     bool readOnly = (bool)dateTimeFormatInfoReadOnly.GetValue(info);
  9.     if (readOnly)
  10.     {
  11.         dateTimeFormatInfoReadOnly.SetValue(info, false);
  12.     }
  13.     if (UsePersianCalendar)
  14.     {
  15.         dateTimeFormatInfoCalendar.SetValue(info, new PersianCalendar());
  16.     }
  17.     info.AbbreviatedDayNames = new string[] { "ی", "I", "س", "چ", "پ", "ج", "O" };
  18.     info.ShortestDayNames = new string[] { "ی", "I", "س", "چ", "پ", "ج", "O" };
  19.     info.DayNames = new string[] { "یکOنEه", "IوOنEه", "ﺳﻪOنEه", "چهCرOنEه", "پنجOنEه", "جمعه", "OنEه" };
  20.     info.AbbreviatedMonthNames = new string[] { "فرورIین", "CرIيEهOE", "IرICI", "Eير", "مرICI", "Oهریور", "مهر", "AECن", "Aذر", "Iی", "Eهمن", "CسفنI", "" };
  21.     info.MonthNames = new string[] { "فرورIین", "CرIيEهOE", "IرICI", "Eير", "مرICI", "Oهریور", "مهر", "AECن", "Aذر", "Iی", "Eهمن", "CسفنI", "" };
  22.     info.AMDesignator = "ق.U";
  23.     info.PMDesignator = "E.U";
  24.     info.FirstDayOfWeek = DayOfWeek.Saturday;
  25.     info.FullDateTimePattern = "yyyy MMMM dddd, dd HH:mm:ss";
  26.     info.LongDatePattern = "yyyy MMMM dddd, dd";
  27.     info.ShortDatePattern = "yyyy/MM/dd";
  28.     if (readOnly)
  29.     {
  30.         dateTimeFormatInfoReadOnly.SetValue(info, true);
  31.     }
  32. }

 

This will fix the DateFormatInfo for Persian Calendar and also months and day names.

Fixing Optional Calendars

An alternative and also more challenging approach would be adding Persian Calendar as an optional calendar. This requires more detail information around how locale specific information are managed by CultureInfo. In fact CultureInfo retrieves culture data from complicated data structures stored in locale files under Windows operating system. Data such as the array of optional calendars are stored in specific data structure and retrieved by special manipulation of pointers.  Following code shows how OptionalCalendars are retrieved from a CultureTableRecord class

internal int[] OptionalCalendars
{
    get
    {
        if (this.optionalCalendars == null)
        {
            this.optionalCalendars = this.m_cultureTableRecord.IOPTIONALCALENDARS;
        }
        return this.optionalCalendars;
    }
}

CultureTableRecord then returns

internal int[] IOPTIONALCALENDARS
{
    get
    {
        return this.GetWordArray(this.m_pData.waCalendars);
    }
}

Which finally returns optional calendars as:

private unsafe int[] GetWordArray(uint iData)
{
    if (iData == 0)
    {
        return new int[0];
    }
    ushort* numPtr = this.m_pPool + ((ushort*) iData);
    int num = numPtr[0];
    int[] numArray = new int[num];
    numPtr++;
    for (int i = 0; i < num; i++)
    {
        numArray[i] = numPtr[i];
    }
    return numArray;
}

Note how pointer calculations are encountered in this evaluation.

To fix the optional calendars of Persian locale one should set the Persian calendar identifier in the appropriate place in the locale data structure. This location may be back calculated from source code above. Then using reflection again to get access to private fields one may get access to the array of optional calendars and fix it on fly.

But there is still another problem. The array lies in a protected memory area. That is you have no write access to that part of memory. A workaround is using VirtualProtect to make this memory writeable before attempting to write back the optional calendars back:

 

Code Snippet
  1. public static  CultureInfo FixOptionalCalendars(CultureInfo culture, int CalenadrIndex)
  2. {
  3.     InvokeHelper ivCultureInfo = new InvokeHelper(culture);
  4.     InvokeHelper ivTableRecord = new InvokeHelper(ivCultureInfo.GetField("m_cultureTableRecord"));
  5.     // Get the m_pData pointer as *void
  6.     System.Reflection.Pointer m_pData = (System.Reflection.Pointer)ivTableRecord.GetField("m_pData");
  7.     ConstructorInfo _intPtrCtor = typeof(IntPtr).GetConstructor(
  8.                     new Type[] { Type.GetType("System.Void*") });
  9.     // Construct a new IntPtr
  10.     IntPtr DataIntPtr = (IntPtr)_intPtrCtor.Invoke(new object[1] { m_pData });
  11.     
  12.     Type TCultureTableData = Type.GetType("System.Globalization.CultureTableData");
  13.     // Convert the Pointer class to object if type CultureTableData to work with
  14.     // reflection API.
  15.     Object oCultureTableData = System.Runtime.InteropServices.Marshal.PtrToStructure(DataIntPtr, TCultureTableData);
  16.     InvokeHelper ivCultureTableData = new InvokeHelper(oCultureTableData);
  17.     // Get waCalendars pointer
  18.     uint waCalendars = (uint)ivCultureTableData.GetField("waCalendars");
  19.     object IOPTIONALCALENDARS = ivTableRecord.GetProperty("IOPTIONALCALENDARS");
  20.  
  21.     // Get m_Pool pointer
  22.     System.Reflection.Pointer m_pool = (System.Reflection.Pointer)ivTableRecord.GetField("m_pPool");
  23.  
  24.     IntPtr PoolInPtr = (IntPtr)_intPtrCtor.Invoke(new object[1] { m_pool });
  25.     // Add the waCalendars offset to pool pointer
  26.     IntPtr shortArrayPtr = new IntPtr((PoolInPtr.ToInt64() + waCalendars*sizeof(ushort)));
  27.     short[] shortArray = new short[1];
  28.     // Now shortArray points to an arry of short integers.
  29.     // Go to read the first value which is the number of elements.
  30.     // Marshal array to read elements.
  31.     System.Runtime.InteropServices.Marshal.Copy(shortArrayPtr, shortArray, 0, 1);
  32.     // shortArray[0] is the number of optional calendars.
  33.     short[] calArray = new short[shortArray[0]];
  34.     // Add one element of short type to point to array of calendars
  35.     IntPtr calArrayPtr = new IntPtr(shortArrayPtr.ToInt64() + sizeof(short));
  36.     // Finally read the array
  37.     System.Runtime.InteropServices.Marshal.Copy(calArrayPtr, calArray, 0, shortArray[0]);
  38.  
  39.     uint old;
  40.     VirtualProtect(calArrayPtr, 100, 0x4, out old);
  41.     calArray[CalenadrIndex] = 0x16;
  42.     System.Runtime.InteropServices.Marshal.Copy(calArray, 0, calArrayPtr, calArray.Length);
  43.     VirtualProtect(calArrayPtr, 100, old, out old);
  44.  
  45.     return culture;
  46.  
  47.  
  48.  
  49. }

 

CultureData in .Net framework 4.0

The CultureTableRecord class has been replaced by CultureData which holds the Optional Calendars as a private array of integers in waCalendars field. This makes correction of Optional Calndars as easy as correcting a private field:

private static CultureInfo _FixOptionalCalendars4(CultureInfo culture, int CalenadrIndex)
{
    FieldInfo cultureDataField = typeof(CultureInfo).GetField("m_cultureData",
         BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance );
    Object cultureData = cultureDataField.GetValue(culture);
    FieldInfo waCalendarsField = cultureData.GetType().GetField("waCalendars",
        BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance);
    int[] waCalendars = (int[])waCalendarsField.GetValue(cultureData);
    if (CalenadrIndex >= 0 && CalenadrIndex < waCalendars.Length)
        waCalendars[CalenadrIndex] = 0x16;
    waCalendarsField.SetValue(cultureData, waCalendars); 
    return culture;
}

Conclusion

Problems with Persian culture in .Net are discussed and methods for correcting these problems are proposed. You may download the sample code from here: Downlad Sample Code

Written by Babak Mahmoudi

August 22, 2011 at 11:09 am

Posted in Persian Localization

Tagged with