the "recurrent_directory_iterator" inside std::filesystem Doesn't support files with "Unicode Characters"

[a1kh]

I do believe that the problem is with the way "std::recursive_directory_iterator" handles the files.
I tried to only print the files paths from the iterator with :

cout << file.path().string();

and it printed the files until it reached a file with Unicode Characters and crashed.
I tried :

cout << file.path().u8string()

.. it didn't crash , but the Unicode Characters are messed up completely.
I tried :

wcout << file.path().u8string()

.. no luck
I tried :

wcout << file.path().u16string()
wcout << file.path().u32string()

.. u16string() and u32string() doesn't exists
the error :
Unhandled exception at 0x00007FFB078C3E49 in ConsoleApplication1.exe: Microsoft C++ exception: std::system_error at memory location 0x000000DD992FE8D8.
here in filesystem :
[[noreturn]] inline void _Throw_system_error_from_std_win_error(const __std_win_error _Errno) {
_THROW(system_error{_Make_ec(_Errno)});
}

I forgot also to mention that i tried "std::wcout << file.path().wstring();" yesterday, but that didn't work too cause it skipped the Unicode files all together .


Specs :
Windows 10. Visual Studio 2019 and the c++17 standard, with everything on default.

[dutch]

What happens if you just cout << file.path() << '\n'

[a1kh]

Program crashes with error :

Unhandled exception at 0x00007FFB078C3E49 in ConsoleApplication1.exe: Microsoft C++ exception: std::system_error at memory location 0x0000003D56B6E6E8.

the error function is :

[[noreturn]] inline void _Throw_system_error_from_std_win_error(const __std_win_error _Errno) {
_THROW(system_error{_Make_ec(_Errno)});
}

the function is in file :
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\include\filesystem

[dutch]

Post a small complete program that replicates the error.

[a1kh]

#include <iostream>
#include <filesystem>
#include <vector>
#include <string>
#include <string.h>
namespace fs = std::filesystem;
using namespace std;

int main() try // -------------- MAIN -------------
{
std::string mainFolder = "F:/testMp4";

cout << "this is hte cout from boost iterator" << endl;
for (auto& file : fs::recursive_directory_iterator(mainFolder))
{
cout << "testing ......................." << '\n';
cout << file.path() << '\n';
//std::wcout << file.path().wstring();
//cout << file.path().u8string() << endl;

// cout << endl;
};
//


}
catch (const fs::filesystem_error& e) {
std::cout << e.what() << '\n';
}

[a1kh]

You can try it by putting some files in a folder, the files names should have "Unicode Characters". It will crash

[dutch]

So the simplest program that reproduces your error is

#include <iostream>
#include <filesystem>
namespace fs = std::filesystem;

int main() try
{
    for (auto& file : fs::recursive_directory_iterator("adir"))
        std::cout << file.path() << '\n';
}
catch (const std::exception& e) {
    std::cout << e.what() << '\n';
}

It works fine for me, but I'm on linux, which uses UTF-8, whereas windows (AFAIK) uses UTF-16 (although I thought they were updating to UTF-8 but I don't know).

[seeplus]

Are you using VS? To what is project/properties/configuration/advanced/Character-Set set?

[a1kh]

dutch (2035):
yes i am on windows , do you believe that there is a way to fix this ?

seeplus (1145):

the setting is : Use Unicode Character Set

[jlb]

What is the exact version of the compiler?

What is the exact version of Windows?

Edit: Also have you tried the method used in the following documentation?

https://docs.microsoft.com/en-us/cpp/standard-library/file-system-navigation?view=msvc-160

Also when you say "it crashed" what was the exact error message?

[a1kh]

jlb (4852):

Sorry I forgot to write those information : Windows 10. Visual Studio 2019 and the c++17 standard, with everything on default.

the console crashes , and Visual Studio takes me to filesystem file , then to shows the error messages. All the error messages are in the first post.

Could you give me a simple code for the method the you meant, with the includes, so that I can understand clearly. In the link there is no iterator code, and the links inside it takes me to the normal "recursive_directory_iterator" description without any examples.

[jlb]

The link I pointed out has code that is showing you how to use Windows Unicode features with their std::filesystem code. Since you say the program is crashing because of Unicode characters in the directory structure you should look at that code for hints on how you need to handle the construction of the "path" and how to print out the directory contents that contain possible Windows Unicode characters.

By the way I would expect that you should be able to trace the "crash" back to somewhere in your code.

[dutch]

How about something like this:

#include <iostream>
#include <filesystem>
#include <string>
namespace fs = std::filesystem;

int main() try
{
    std::wstring d = L"adir";
    for (auto& file: fs::recursive_directory_iterator(d))
        std::wcout << file.path() << L'\n';
}
catch (const std::exception& e) {
    std::wcout << e.what() << L'\n';
}

[jlb]

By the way while your code doesn't "crash the console" for me it does throw an exception: "filesystem error: cannot increment recursive directory iterator: Permission denied" but only when running the program as a "normal user", if I run the program as "root" it runs to proper termination.

In case you haven't guessed I'm using Linux, not Windows.

[a1kh]

jlb (4854):
Sorry I am a beginner, I couldn't understand what they have on that link clearly.

handle the construction of the "path" and how to print out the directory contents that contain possible Windows Unicode characters.

This idea sounds great. I hope that you can help to achieve promising results to solve this problem, cause really I can't totally understand how to implement it.

As for Windows, yeah . I totally understand what you mean. I tried the code the day before yesterday on another drive other than C, just to eliminate the "Windows Permissions" problem too. no luck.

I am an "Internal Administrator", so permissions are not a problem for me for 99% of time.

[dutch]

Did you try the code in my last post?

[a1kh]

dutch (2036):
Unfortunately it didn't work, it skipped all the Unicode files.

[jlb]

If you're such a beginner, what is the reason you want to try to recursively iterate trough your directories?

But as for giving you much more information, you're probably going to need to wait for a Windows expert that can either duplicate the problem or offer further direction. Since I haven't touched any Windows operating system this century I can't offer much further advice.

[a1kh]

jlb (4855):
Thank you.I hope that i could find a solution for this problem.
I am trying to iterate trough directories to collect files and, push them back to a vector, as a practice project.

Then maybe get the size of each file, add it to some kind of a database, and so on. Practice makes perfect :-p .