Skip to main content

Opening files with unicode (japanese/chinese) characters in filename using Perl

(Read a complete article on the matter and much more
"Unicode issues regarding the Window OS file system and their handling from Perl)

Currently there is no way to manipulate a file named using Unicode characters,by using Perl's built in functions.
The perl 5.10 todo wish list states that functions like chdir, opendir, readdir, readlink, rename, rmdir e.g
"could potentially accept Unicode filenames either as input or output".
Windows default encoding is UTF-16LE,but the console 'dir' command will only return ANSI names.Thus unicode characters are replaced with "?"
,even if you invoke the console using the unicode switch (cmd.exe /u),change the codepage to 65001 which is utf8 on windows and use lucida console true type font which supports unicode.
A workaround is to use the COM facilities provided by windows (in this case Scripting.FileSystemObject) which provide a much higher level of abstraction or use the Win32 api calls.
I tried to read a file with japanese characters in the filename which resides in the current folder and then move the file to another folder.
The filename is "は群馬県高崎市を拠点に、様々なメディ.txt"
Since opendir ,readdir,rename etc do not support unicode you have to reside to the Scripting.FileSystemObject methods and properties which accept unicode.

This is the actual code which moves all files with .txt extension:

use Win32::OLE qw(in);
use Devel::Peek;

#CP_UTF8 is very important as it translates between Perl strings and U
+nicode strings used by the OLE interface

Win32::OLE->Option(CP => Win32::OLE::CP_UTF8);  

$obj = Win32::OLE->new('Scripting.FileSystemObject');

$folder = $obj->GetFolder(".");

$collection= $folder->{Files};

mkdir ("c:\\newfolder")||die;

foreach $value (in $collection) {
$filename= %$value->{Name};
next if ($filename !~ /.txt/);
Dump("$filename");  #check if the utf8 flag is on
$file=$obj->GetFile("$filename"); 
$file->Move("c:\\newfolder\\$filename");
print Win32::OLE->LastError() || "success\n\n";
}

If all goes well then a new folder -suprsingly- called 'newfolder' in C: drive would have been created and the japanese named filed should been have moved there.

This will only work if you have the asian languages (regional setings) support enabled and you should be able to see the japanase name in explorer as above

Comments

Anonymous said…
Thanks a lot,saved me loads of time.Exactly what I needed!
Daku said…
Spent ages trying to rename a file using Win32::Unicode / Win32API::File with no luck.
This worked without any hassle, thanks :)

Popular posts from this blog

Open Source Is Not Just About Software

  It's about infrastructure as well, something that although not attracting the limelight, is as important as the open source software it hosts. Today the stewards of the largest open source registries in the world have issued an open letter demanding urgent reform in how open source infrastructure is funded, maintained, and operated.  https://www.i-programmer.info/news/136-open-source/18334-open-source-is-not-just-about-software.html

Europe Gets Its Own LLM

  EuroLLM is a fully open-sourced large language model made in Europe and built to support all twenty-four official EU languages. While several European states have separately produced their own LLMs, such as Greece's "Meltemi" or the recent offering from Switzerland "Apertus", there was no solution that catered for the languages of all 24 states belonging to the EU block. The time has come for this status to change with the appearance of EuroLLM, a foundational model that supports them all. https://www.i-programmer.info/news/105-artificial-intelligence/18448-europe-gets-its-own-llm.html

With MCP Docs Servers You'll Never Run Out Of Fresh Documentation

  MCP has changed the way you interact with your tools overnight. Now it targets your documentation. Wouldn't be great to have the latest and updated code samples and documentation of your favorite framework and libraries ready at your fingertips? Plus, be able to talk to it in natural language? https://www.i-programmer.info/news/90-tools/18248-with-mcp-docs-servers-youll-never-run-out-of-fresh-documentation-.html