Using iconv for Utf8Converter on Apple device by jchen351 · Pull Request #15599 · microsoft/onnxruntime

jchen351 · 2023-04-20T06:45:15Z

Description

using iconv for Utf8Converter on Apple device

Motivation and Context

Because wstring_convert will be deprecated in macOS SDK 13.3 C++17

…d in macOS SDK 13.3 C++17

snnn · 2023-04-26T04:01:19Z

 		DBDB57DB2603211A004F16BE /* Info.plist */ = {isa = PBXFileReference; lastKnownFileType = text.plist.xml; path = Info.plist; sourceTree = "<group>"; };
 		DBDB588A2609B18F004F16BE /* Resources */ = {isa = PBXFileReference; lastKnownFileType = folder; name = Resources; path = OnnxruntimeModuleTest/Resources; sourceTree = "<group>"; };
 		DBDB58AF262A92D6004F16BE /* OnnxruntimeModuleTest.mm */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.objcpp; path = OnnxruntimeModuleTest.mm; sourceTree = "<group>"; };
+		E3D7A60D29F8D87D00FF5265 /* libiconv.tbd */ = {isa = PBXFileReference; lastKnownFileType = "sourcecode.text-based-dylib-definition"; name = libiconv.tbd; path = Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/lib/libiconv.tbd; sourceTree = DEVELOPER_DIR; };


Where can we find the license of this file: Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/lib/libiconv.tbd ?

Would it be bad to hardcode a XCode version here?

This file was generated from XCode GUI Application. I have to link the library for these 2 xcode projects from the Xcode GUI. The library is part of Apple SDK.

I can try to remove the 13.3 from the MacOSX13.3.sdk, and it should works.

The library must come with some headers files. One of them should be iconv.h. Could you please open the file and inspect its license?

/* Copyright (C) 1999-2003, 2005-2006 Free Software Foundation, Inc.
This file is part of the GNU LIBICONV Library.

The GNU LIBICONV Library is free software; you can redistribute it
and/or modify it under the terms of the GNU Library General Public
License as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

The GNU LIBICONV Library is distributed in the hope that it will be
useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Library General Public License for more details.

You should have received a copy of the GNU Library General Public
License along with the GNU LIBICONV Library; see the file COPYING.LIB.
If not, write to the Free Software Foundation, Inc., 51 Franklin Street,
Fifth Floor, Boston, MA 02110-1301, USA. */

It is GNU, but is the copyright expired?

No. I don't think so. We should avoid using it.

Then we have to find another alternative before we adapt MacOS SDK 13.3.

See this: https://stackoverflow.com/questions/57734434/libiconv-or-iconv-undefined-symbol-on-mac-osx/57734435

snnn · 2023-04-26T15:07:27Z

@skottmckay, what do you think?

skottmckay · 2023-05-16T23:50:05Z

Probably safest to ask CELA.

I don't recall having to worry about licenses for any other libraries that are part of the OS (if we're using the /usr/lib version). Is there any reason why the /usr/lib version would not be sufficient?

Also not clear to me why it would be okay to use libiconv on Linux but not on mac, but if we can stick to the /usr/lib version maybe that question doesn't matter.

snnn · 2023-05-16T23:56:19Z

We do not use libiconv on Linux. libiconv is from GNU. https://www.gnu.org/software/libiconv/ . We use glibc on Linux. glibc is not licensed under GPL.

skottmckay · 2023-05-17T09:05:08Z

The stackoverflow article made it sound like the default iconv in macos was not the GNU libiconv one, but the license in the SDK for MacOSX (/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/iconv.h) is the GNU license.

I can only find iconv.h in the Xcode SDKs. No other locations.

// find . -name iconv.h -type f 2>&/dev/null
./System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/include/iconv.h
./System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include/iconv.h
./System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/include/iconv.h
./System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/iconv.h
./System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator.sdk/usr/include/iconv.h
./System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/iconv.h
./System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include/iconv.h
./Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/include/iconv.h
./Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include/iconv.h
./Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/include/iconv.h
./Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/iconv.h
./Applications/Xcode.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator.sdk/usr/include/iconv.h
./Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/iconv.h
./Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include/iconv.h

https://opensource.apple.com/source/libiconv/libiconv-9/libiconv/include/iconv.h.in.auto.html

Not sure if macos doesn't use libiconv so doesn't care about the GPL license or what.

Can we use the source from the gnu libc version or are there too many files to copy?

Other option might be boost::locale

edgchen1 · 2023-05-18T23:48:51Z

from here: https://www.gnu.org/software/libiconv/

Copyright
The libiconv and libcharset libraries and their header files are under LGPL.
The iconv program is under GPL.

is LGPL ok for us to use?

snnn · 2023-05-19T01:17:08Z

is LGPL ok for us to use?

If we dynamically link to it, not much obligation. Just make sure to not include it in our packages.

snnn

Based on an internal discussion, we have a better choice than this library. We can use the system's builtin APIs.

# Conflicts: # onnxruntime/core/providers/cpu/nn/string_normalizer.cc

Changming Sun and others added 4 commits April 19, 2023 23:38

Fix iconv link issue

9ac2e11

Merge remote-tracking branch 'origin/main' into snnn/iconv

e6fac75

using iconv for Utf8Converter since wstring_convert will be deprecate…

0aa1b74

…d in macOS SDK 13.3 C++17

Update CMakeLists.txt

2b4e744

jchen351 changed the title ~~Cjian/iconv mac~~ Using iconv for Utf8Converter on Apple device Apr 20, 2023

jchen351 marked this pull request as ready for review April 20, 2023 06:46

Changming Sun and others added 7 commits April 19, 2023 23:49

Merge remote-tracking branch 'origin/main' into snnn/iconv

5ea935e

Merge remote-tracking branch 'origin/main' into snnn/iconv

b7fb447

update

e9ababb

Merge branch 'main' into Cjian/iconv_mac

f341487

Merge remote-tracking branch 'origin/snnn/iconv' into Cjian/iconv_mac

2aa324a

Merge branch 'main' into Cjian/iconv_mac

9c3550f

adding iconv to ios test packages build

a844ec7

snnn previously approved these changes Apr 25, 2023

View reviewed changes

jchen351 added 2 commits April 25, 2023 20:57

Merge branch 'main' into Cjian/iconv_mac

a20a0f6

adding iconv to react ios test

ca0a8e2

jchen351 dismissed snnn’s stale review via ca0a8e2 April 26, 2023 03:57

snnn reviewed Apr 26, 2023

View reviewed changes

snnn requested a review from skottmckay April 26, 2023 04:02

Remove 13.3 from MacOSX

5b0f73d

snnn mentioned this pull request May 15, 2023

Fix some build issues on MacOS with Xcode 14.3. #15878

Merged

snnn suggested changes May 23, 2023

View reviewed changes

jchen351 added 2 commits September 25, 2023 13:30

Rolling back to main

9ae4c4a

Merge branch 'main' into Cjian/iconv_mac

8a44a12

# Conflicts: # onnxruntime/core/providers/cpu/nn/string_normalizer.cc

jchen351 closed this Nov 17, 2023

Uh oh!

Conversation

jchen351 commented Apr 20, 2023

Description

Motivation and Context

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snnn commented Apr 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

skottmckay commented May 16, 2023

Uh oh!

snnn commented May 16, 2023

Uh oh!

skottmckay commented May 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

edgchen1 commented May 18, 2023

Uh oh!

snnn commented May 19, 2023

Uh oh!

snnn left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

snnn commented Apr 26, 2023 •

edited

Loading

skottmckay commented May 17, 2023 •

edited

Loading

snnn left a comment •

edited

Loading