I am trying to run a python program on a data processing (linux) cluster. The program runs fine if I ssh to the cluster from my linux desktop and start it but fails if I ssh to the cluster from my mac laptop and try to run it I get the following error:

[muzzle@cluster]~% acme-calibrate detector DARK --in-folder /gpfs/acme/exp/instrument/201802/experiment1/raw --out-folder /home/muzzle/dark_out --run 28
Traceback (most recent call last):
  File "/home/boff/calframework/framework/extern/bin/acme-calibrate", line 11, in <module>
    load_entry_point('acme-Offline-Calibration', 'console_scripts', 'acme-calibrate')()
  File "/home/boff/calframework/framework/extern/lib/python3.4/site-packages/setuptools-39.1.0-py3.4.egg/pkg_resources/__init__.py", line 480, in load_entry_point
  File "/home/boff/calframework/framework/extern/lib/python3.4/site-packages/setuptools-39.1.0-py3.4.egg/pkg_resources/__init__.py", line 2691, in load_entry_point
  File "/home/boff/calframework/framework/extern/lib/python3.4/site-packages/setuptools-39.1.0-py3.4.egg/pkg_resources/__init__.py", line 2322, in load
  File "/home/boff/calframework/framework/extern/lib/python3.4/site-packages/setuptools-39.1.0-py3.4.egg/pkg_resources/__init__.py", line 2328, in resolve
  File "/home/boff/pycalibrate_tmp/acme_calibrate/calibrate.py", line 236, in <module>
nb = nbformat.read(f, as_version=4)
  File "/home/boff/calframework/framework/extern/lib/python3.4/site-packages/nbformat/__init__.py", line 141, in read
return reads(fp.read(), as_version, **kwargs)
  File "/home/boff/calframework/framework/extern/lib/python3.4/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 150: ordinal not in range(128)

It seems to be related to some conversion from UTF to ASCII, but I'm out of my depth.

I get the same error with both terminal.app and iterm2. Btw I used to have the iterm2 shell integration installed but I have removed them now.

The locale on the laptop is:

$ locale
LANG=
LC_COLLATE="C"
LC_CTYPE="UTF-8"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

The locale on the remote server is:

% locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=en_US.UTF-8
LC_CTYPE=UTF-8
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

This is the locale on my linux desktop, it is identical to that of the cluster:

$ locale
LANG=en_GB.UTF-8
LANGUAGE=en_GB:en
LC_CTYPE="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_PAPER="en_GB.UTF-8"
LC_NAME="en_GB.UTF-8"
LC_ADDRESS="en_GB.UTF-8"
LC_TELEPHONE="en_GB.UTF-8"
LC_MEASUREMENT="en_GB.UTF-8"
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=
  • Do both terminals use UTF-8? What are the LANG and LC_* variables in both cases? See export | grep -E ' (LANG|LC_)' – pabouk Dec 7 at 9:14
  • It looks to me like they do. I have added the output of locale to the question, if you want to take a look – muzzle Dec 7 at 11:40

It looks like the correct solution was to unset "Set locale environment variables on startup" in the terminal advanced options as described in https://blog.remibergsma.com/2012/07/10/setting-locales-correctly-on-mac-osx-terminal-application/

To do the same thing on iterm2 you have to unselect Preferences > Profiles > Terminal > Set locale variables automatically, as described here https://apple.stackexchange.com/questions/83428/lang-variable-wrong-in-iterm2-wheres-the-bug

The post OS X Terminal UTF-8 issues has some useful advice, based on A guide to help set up Terminal Apps:

  1. In Terminal->Preferences->Settings->Advanced:

    • Under International, ensure the character encoding is set to Unicode (UTF-8).

    • Under Emulation, ensure that Escape non-ASCII input with Control-V is not set.

    This by itself may be enough to solve the problem.

  2. Use the command locale and look at the LC_CTYPE line, which should end with .UTF-8. If it doesn't, then in bash .profile or .bashrc in the home directory, add a line like this:

    export LC_CTYPE=your-locale-here.UTF-8
    
  3. Add the following lines to .inputrc in the home directory (create it if necessary):

    set meta-flag on
    set input-meta on
    set output-meta on
    set convert-meta off
    

You need to restart bash for the new settings to take effect.

  • I just tried all three steps but it did not work. Btw, I have added the output of locale to the question, if you want to take a look – muzzle Dec 7 at 11:41
  • It would be interesting to examine the differences in output of locale for linux and mac. – harrymc Dec 7 at 11:48
  • Hi @harrymc, I just added that to the description of the issue – muzzle Dec 7 at 11:57
  • 1
    Does the locale on the server differ if you connect from the Linux and from the Mac? – pabouk Dec 7 at 12:05
  • The locale on the desktop is identical to that of the cluster. I'll try to set my mac locale to the same – muzzle Dec 7 at 13:16

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.