Possible bugs with string encoding

[pp-mo]

Is this a bug?
My example is pretty simple, if maybe a bit long,
but here it is ...

>>> import netCDF4 as nc
>>> import numpy as np
>>> import sys
>>> 
>>> print("Python version:", sys.version)
Python version: 3.12.12 | packaged by conda-forge | (main, Oct 13 2025, 14:34:15) [GCC 14.3.0]
>>> print("NetCDF4 version:", nc.__version__)
NetCDF4 version: 1.7.2
>>> 
>>> filepath = "tmp.nc"
>>> nx, n_strlen = 3, 10
>>> 
>>> ds = nc.Dataset(filepath, "w")
>>> ds.createDimension("x", nx)
"<class 'netCDF4.Dimension'>": name = 'x', size = 3
>>> ds.createDimension("nstr", n_strlen)
"<class 'netCDF4.Dimension'>": name = 'nstr', size = 10
>>> v = ds.createVariable("v", "S1", dimensions=("x", "nstr",))
>>> 
>>> strings = ['Münster', 'London', 'Amsterdam']
>>> print(
...     "Strings for variable content:"
...     f"\n{strings}"
...     "\n"
... )
Strings for variable content:
['Münster', 'London', 'Amsterdam']

>>> 
>>> bbytes = [text.encode("utf-8") for text in strings]
>>> pad = b'\0' * n_strlen
>>> bbytes = [(x + pad)[:n_strlen] for x in bbytes]
>>> print(
...     "Equal-length byte objects with utf-8 encoding:"
...     f"\n{bbytes}"
...     "\n"
... )
Equal-length byte objects with utf-8 encoding:
[b'M\xc3\xbcnster\x00\x00', b'London\x00\x00\x00\x00', b'Amsterdam\x00']

>>> 
>>> bytesarray = np.array(bbytes, dtype=f"S{n_strlen}")
>>> print(
...     "\nBytes(Sxx) array:"
...     f"\n{bytesarray}"
...     f"\n  :: shape={bytesarray.shape} dtype ={bytesarray.dtype}"
...     "\n"
... )

Bytes(Sxx) array:
[b'M\xc3\xbcnster' b'London' b'Amsterdam']
  :: shape=(3,) dtype =|S10

>>> 
>>> chararray = np.array([[bb[i:i+1] for i in range(n_strlen)] for bb in bbytes])
>>> print(
...     "\nCharacter(S1) array:"
...     f"\n{chararray}"
...     f"  :: shape={chararray.shape} dtype ={chararray.dtype}"
...     "\n"
... )

Character(S1) array:
[[b'M' b'\xc3' b'\xbc' b'n' b's' b't' b'e' b'r' b'' b'']
 [b'L' b'o' b'n' b'd' b'o' b'n' b'' b'' b'' b'']
 [b'A' b'm' b's' b't' b'e' b'r' b'd' b'a' b'm' b'']]  :: shape=(3, 10) dtype =|S1

>>> 
>>> # Store chararray in variable, and mark as UTF8 encoded
>>> v._Encoding = "UTF-8"
>>> v[:] = chararray
>>> 
>>> ds.close()
>>> 
>>> from os import system as run
>>> print("\nNCDUMP of file:")

NCDUMP of file:
>>> run(f"ncdump {filepath}")
netcdf tmp {
dimensions:
	x = 3 ;
	nstr = 10 ;
variables:
	char v(x, nstr) ;
		v:_Encoding = "UTF-8" ;
data:

 v =
  "M\303\274nster",
  "London",
  "Amsterdam" ;
}
0
>>> 
>>> ds2 = nc.Dataset(filepath)
>>> v = ds2.variables['v']
>>> data = v[:]
>>> print(
...     "\nData read back from file variable:"
...     f"\n{data}"
...     f"  :: shape={data.shape} dtype ={data.dtype}"
...     "\n"
... )

Data read back from file variable:
['Münster\x00\x00L' 'ondon\x00\x00\x00\x00A' 'msterdam']  :: shape=(3,) dtype =<U10

>>> 
>>> print(
...     "individual elements..."
...     f"{''.join(f"\n  {elem}" for elem in data)}"
...     "\n"
... )
individual elements...
  MünsterL
  ondonA
  msterdam

>>> 
>>> v.set_auto_chartostring(False)
>>> data = v[:]
>>> 
>>> print(
...     "\nSame variable data, read *without* encoding interpretation:",
...     '\n', data,
...     f"\n  :: shape={data.shape} dtype ={data.dtype}"
...     "\n"
... )

Same variable data, read *without* encoding interpretation: 
 [[b'M' b'\xc3' b'\xbc' b'n' b's' b't' b'e' b'r' -- --]
 [b'L' b'o' b'n' b'd' b'o' b'n' -- -- -- --]
 [b'A' b'm' b's' b't' b'e' b'r' b'd' b'a' b'm' --]] 
  :: shape=(3, 10) dtype =|S1

So what??

The key point is what happens when I try to read back the variable, with "chartostring" enabled:

Data read back from file variable:
['Münster\x00\x00L' 'ondon\x00\x00\x00\x00A' 'msterdam'] :: shape=(3,) dtype =<U10

individual elements...
MünsterL
ondonA
msterdam

So, that does look wrong.
However, when read without decoding, the content is all ok.

Additional

When I try to write the variable, this also hits a problem...

>>> with nc.Dataset(filepath, 'r+') as ds2:
...     print("\nModifying variable:")
...     v = ds2.variables['v']
...     v[0] = "Liége"
... 

Modifying variable:
Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
  File "src/netCDF4/_netCDF4.pyx", line 5513, in netCDF4._netCDF4.Variable.__setitem__
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 2: ordinal not in range(128)
>>> 

So, do I just need to set "_Encoding" to something different?

[jswhit]

I think the problem is that the automatic conversion of char arrays to/from string arrays only works correctly for ascii data (byte arrays), not unicode data (which can be 1-4 bytes per character). This should be made clear in the docs, and perhaps an error or warning raised if you try to do it with anything but _Encoding='ascii'.

EDIT: looking more closely at the code, I don't actually think this is the case - it seems like it should work. More investigation is needed...

[jswhit]

PR #1443 fixes the logic errors in stringtochar and chartostring for unicode strings (previously only worked for ascii)

[pp-mo]

Many thanks for explaining, we can work around this for now.
Also for considering a fix -- I can now see, this is not that simple an issue!