Skip to main content

Opening files with unicode (japanese/chinese) characters in filename using Perl

(Read a complete article on the matter and much more
"Unicode issues regarding the Window OS file system and their handling from Perl)

Currently there is no way to manipulate a file named using Unicode characters,by using Perl's built in functions.
The perl 5.10 todo wish list states that functions like chdir, opendir, readdir, readlink, rename, rmdir e.g
"could potentially accept Unicode filenames either as input or output".
Windows default encoding is UTF-16LE,but the console 'dir' command will only return ANSI names.Thus unicode characters are replaced with "?"
,even if you invoke the console using the unicode switch (cmd.exe /u),change the codepage to 65001 which is utf8 on windows and use lucida console true type font which supports unicode.
A workaround is to use the COM facilities provided by windows (in this case Scripting.FileSystemObject) which provide a much higher level of abstraction or use the Win32 api calls.
I tried to read a file with japanese characters in the filename which resides in the current folder and then move the file to another folder.
The filename is "は群馬県高崎市を拠点に、様々なメディ.txt"
Since opendir ,readdir,rename etc do not support unicode you have to reside to the Scripting.FileSystemObject methods and properties which accept unicode.

This is the actual code which moves all files with .txt extension:

use Win32::OLE qw(in);
use Devel::Peek;

#CP_UTF8 is very important as it translates between Perl strings and U
+nicode strings used by the OLE interface

Win32::OLE->Option(CP => Win32::OLE::CP_UTF8);  

$obj = Win32::OLE->new('Scripting.FileSystemObject');

$folder = $obj->GetFolder(".");

$collection= $folder->{Files};

mkdir ("c:\\newfolder")||die;

foreach $value (in $collection) {
$filename= %$value->{Name};
next if ($filename !~ /.txt/);
Dump("$filename");  #check if the utf8 flag is on
$file=$obj->GetFile("$filename"); 
$file->Move("c:\\newfolder\\$filename");
print Win32::OLE->LastError() || "success\n\n";
}

If all goes well then a new folder -suprsingly- called 'newfolder' in C: drive would have been created and the japanese named filed should been have moved there.

This will only work if you have the asian languages (regional setings) support enabled and you should be able to see the japanase name in explorer as above

Comments

Anonymous said…
Thanks a lot,saved me loads of time.Exactly what I needed!
Daku said…
Spent ages trying to rename a file using Win32::Unicode / Win32API::File with no luck.
This worked without any hassle, thanks :)

Popular posts from this blog

The PaSh Project gives your POSIX script superpowers by utilizing parallelization in order to speed up execution times.

The PaSh Project gives your POSIX script superpowers by utilizing parallelization in order to speed up execution times. This leads to faster results for data scientists,  engineers,  biologists,  economists,  administrators,  and programmers. I remember the time when the saying was "Learn Perl so you don't have to learn the Shell and its hundreds of utilities". Fast forward some decades and the use of shell scripts still has not been eradicated. On the contrary, their use has increased due to the rise of containers, VM's, administering the cloud, and Linux itself. This also serves as a lesson to those who are quick to denounce technologies as 'dead'. There's a time where a new use case revitalizes an old technology. So what is meant by "Unix philosophy"?  full article on i-programmer: https://www.i-programmer.info/news/90-tools/14990-the-pash-project-advancing-the-unix-philosophy-one-step-further.html    

Open Source Is Not Just About Software

  It's about infrastructure as well, something that although not attracting the limelight, is as important as the open source software it hosts. Today the stewards of the largest open source registries in the world have issued an open letter demanding urgent reform in how open source infrastructure is funded, maintained, and operated.  https://www.i-programmer.info/news/136-open-source/18334-open-source-is-not-just-about-software.html

DeepSWE - Best Benchmark For Evaluating AI Coding Agents?

  DeepSWE is a high-fidelity benchmarking platform designed to evaluate the performance of frontier AI coding agents on complex long-horizon software engineering tasks. Unlike traditional benchmarks that reuse existing GitHub data, this system utilizes original, contamination-free tasks across five major programming languages to prevent models from relying on memorized solutions, creating a much clearer separation between frontier coding models than older public benchmarks like SWE-bench. https://www.i-programmer.info/news/105-artificial-intelligence/19016-deepswe-best-benchmark-for-evaluating-ai-coding-agents.html