CLI shows question marks instead of Russian symbols

[alesana-san]

Hi!

What started in a Slack chat grew into this issue :)

I have a test with Russian words in its name. So I expect it to be "Russian" when I call it via cli.

Here's my preset:

• NLS_DATABASE_PARAMETERS:
  • NLS_TERRITORY = AMERICA
  • NLS_LANGUAGE = AMERICAN
  • NLS_CHARACTERSET = CL8ISO8859P5
• NLS_SESSION_PARAMETERS (via DataGrip):
  • NLS_LANGUAGE = RUSSIAN
  • NLS_TERRITORY = RUSSIA
• Database Version = 12.2.0.1.0

Calling ut.run in DataGrip results in a good output with all my Russian symbols.

Okay, let's proceed with cli:

C:\Users\Shiryaev1-RY>chcp
Текущая кодовая страница: 866

C:\Users\Shiryaev1-RY>utplsql.bat info "some_user"/"q1w2e3r4t5y6u7i8o9p0"@local_pdb
utPLSQL-cli 3.1.8-SNAPSHOT.local
utPLSQL-java-api 3.1.8.546
utPLSQL 3.1.6.2735

Calling as it is:

C:\Users\Shiryaev1-RY>utplsql.bat run "some_user"/"q1w2e3r4t5y6u7i8o9p0"@local_pdb -p=SP.ut_sp_hand_get_reqs.basic_usage -d
2019-11-26 17:54:08 [main] INFO  org.utplsql.cli.RunAction - ############# utPLSQL cli ##############
#                                      #
#   utPLSQL-cli 3.1.8-SNAPSHOT.local   #
#   utPLSQL-java-api 3.1.8.546         #
#   Java-Version: 1.8.0_221            #
#   ORACLE_HOME: null                  #
#   NLS_LANG: null                     #
#                                      #
#   Thanks for testing!                #
#                                      #
########################################
2019-11-26 17:54:08 [main] INFO  org.utplsql.cli.RunAction -
2019-11-26 17:54:09 [main] INFO  o.u.c.d.TestedDataSourceProvider - Use connection string jdbc:oracle:thin:****/****@local_pdb
2019-11-26 17:54:09 [main] INFO  org.utplsql.cli.RunAction - Successfully connected to database. UtPLSQL core: v3.1.6.2735
2019-11-26 17:54:09 [main] INFO  org.utplsql.cli.RunAction - Oracle-Version: 12.2.0.1.0
2019-11-26 17:54:10 [main] DEBUG org.utplsql.api.reporter.Reporter - Database-reporter initialized, Type: UT_DOCUMENTATION_REPORTER, ID: 98420D7BA25A031BE053020012ACA88D
2019-11-26 17:54:10 [pool-1-thread-1] INFO  org.utplsql.cli.RunTestRunnerTask - Running tests now.
2019-11-26 17:54:10 [pool-1-thread-1] INFO  org.utplsql.cli.RunTestRunnerTask - --------------------------------------
2019-11-26 17:54:10 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - TestRunner initialized
2019-11-26 17:54:10 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Running on utPLSQL v3.1.6.2735
2019-11-26 17:54:10 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Initializing reporters
2019-11-26 17:54:11 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Running tests
simple_payment
  sp_req_handler_pkg
2019-11-26 17:54:11 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Running tests finished.
    get_reqs_to_process
      ????? ????????? sp_req_handler_pkg.get_reqs_to_process
        ???? ???????? ????????????? [,004 sec]

Finished in ,028223 seconds
1 tests, 0 failed, 0 errored, 0 disabled, 0 warning(s)

2019-11-26 17:54:11 [main] INFO  org.utplsql.cli.RunAction - --------------------------------------
2019-11-26 17:54:11 [main] INFO  org.utplsql.cli.RunAction - All tests done.

Nope, didn't work.

Let's try with LC_ALL:

C:\Users\Shiryaev1-RY>set LC_ALL=ru_RU.utf-8

C:\Users\Shiryaev1-RY>utplsql.bat run "some_user"/"q1w2e3r4t5y6u7i8o9p0"@local_pdb -p=SP.ut_sp_hand_get_reqs.basic_usage -d
2019-11-26 17:55:30 [main] INFO  org.utplsql.cli.RunAction - ############# utPLSQL cli ##############
#                                      #
#   utPLSQL-cli 3.1.8-SNAPSHOT.local   #
#   utPLSQL-java-api 3.1.8.546         #
#   Java-Version: 1.8.0_221            #
#   ORACLE_HOME: null                  #
#   NLS_LANG: null                     #
#                                      #
#   Thanks for testing!                #
#                                      #
########################################
2019-11-26 17:55:30 [main] INFO  org.utplsql.cli.RunAction -
2019-11-26 17:55:30 [main] INFO  o.u.c.d.TestedDataSourceProvider - Use connection string jdbc:oracle:thin:****/****@local_pdb
2019-11-26 17:55:30 [main] INFO  org.utplsql.cli.RunAction - Successfully connected to database. UtPLSQL core: v3.1.6.2735
2019-11-26 17:55:30 [main] INFO  org.utplsql.cli.RunAction - Oracle-Version: 12.2.0.1.0
2019-11-26 17:55:31 [main] DEBUG org.utplsql.api.reporter.Reporter - Database-reporter initialized, Type: UT_DOCUMENTATION_REPORTER, ID: 9842124F42B6034AE053020012ACF6A3
2019-11-26 17:55:31 [pool-1-thread-1] INFO  org.utplsql.cli.RunTestRunnerTask - Running tests now.
2019-11-26 17:55:31 [pool-1-thread-1] INFO  org.utplsql.cli.RunTestRunnerTask - --------------------------------------
2019-11-26 17:55:31 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - TestRunner initialized
2019-11-26 17:55:31 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Running on utPLSQL v3.1.6.2735
2019-11-26 17:55:31 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Initializing reporters
2019-11-26 17:55:31 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Running tests
2019-11-26 17:55:31 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Running tests finished.
simple_payment
  sp_req_handler_pkg
    get_reqs_to_process
      ????? ????????? sp_req_handler_pkg.get_reqs_to_process
        ???? ???????? ????????????? [,004 sec]

Finished in ,009586 seconds
1 tests, 0 failed, 0 errored, 0 disabled, 0 warning(s)

2019-11-26 17:55:31 [main] INFO  org.utplsql.cli.RunAction - --------------------------------------
2019-11-26 17:55:31 [main] INFO  org.utplsql.cli.RunAction - All tests done.

Noooooooope. Okay, let's try with NLS_LANG:

C:\Users\Shiryaev1-RY>set LC_ALL=

C:\Users\Shiryaev1-RY>set NLS_LANG=russian_russia.CL8ISO8859P5

C:\Users\Shiryaev1-RY>utplsql.bat run "some_user"/"q1w2e3r4t5y6u7i8o9p0"@local_pdb -p=SP.ut_sp_hand_get_reqs.basic_usage -d
2019-11-26 17:57:39 [main] INFO  org.utplsql.cli.RunAction - ################ utPLSQL cli ################
#                                           #
#   utPLSQL-cli 3.1.8-SNAPSHOT.local        #
#   utPLSQL-java-api 3.1.8.546              #
#   Java-Version: 1.8.0_221                 #
#   ORACLE_HOME: null                       #
#   NLS_LANG: russian_russia.CL8ISO8859P5   #
#                                           #
#   Thanks for testing!                     #
#                                           #
#############################################
2019-11-26 17:57:39 [main] INFO  org.utplsql.cli.RunAction -
2019-11-26 17:57:39 [main] DEBUG o.u.c.d.TestedDataSourceProvider - NLS settings: BEGIN
EXECUTE IMMEDIATE q'[ALTER SESSION SET NLS_LANGUAGE='russian']';
EXECUTE IMMEDIATE q'[ALTER SESSION SET NLS_TERRITORY='russia']';
END;
2019-11-26 17:57:40 [main] INFO  o.u.c.d.TestedDataSourceProvider - Use connection string jdbc:oracle:thin:****/****@local_pdb
2019-11-26 17:57:40 [main] INFO  org.utplsql.cli.RunAction - Successfully connected to database. UtPLSQL core: v3.1.6.2735
2019-11-26 17:57:40 [main] INFO  org.utplsql.cli.RunAction - Oracle-Version: 12.2.0.1.0
2019-11-26 17:57:40 [main] DEBUG org.utplsql.api.reporter.Reporter - Database-reporter initialized, Type: UT_DOCUMENTATION_REPORTER, ID: 98421A084170039EE053020012AC52B2
2019-11-26 17:57:40 [pool-1-thread-1] INFO  org.utplsql.cli.RunTestRunnerTask - Running tests now.
2019-11-26 17:57:40 [pool-1-thread-1] INFO  org.utplsql.cli.RunTestRunnerTask - --------------------------------------
2019-11-26 17:57:40 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - TestRunner initialized
2019-11-26 17:57:40 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Running on utPLSQL v3.1.6.2735
2019-11-26 17:57:40 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Initializing reporters
2019-11-26 17:57:40 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Running tests
2019-11-26 17:57:40 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Running tests finished.
simple_payment
  sp_req_handler_pkg
    get_reqs_to_process
      ????? ????????? sp_req_handler_pkg.get_reqs_to_process
        ???? ???????? ????????????? [,013 sec]

Finished in ,015801 seconds
1 tests, 0 failed, 0 errored, 0 disabled, 0 warning(s)

2019-11-26 17:57:40 [main] INFO  org.utplsql.cli.RunAction - --------------------------------------
2019-11-26 17:57:40 [main] INFO  org.utplsql.cli.RunAction - All tests done.

God, plz! After looking at Oracle NLS stories I decided to do the following:

C:\Users\Shiryaev1-RY>set NLS_LANG=russian_russia.RU8PC866

C:\Users\Shiryaev1-RY>utplsql.bat run "some_user"/"q1w2e3r4t5y6u7i8o9p0"@local_pdb -p=SP.ut_sp_hand_get_reqs.basic_usage -d
2019-11-26 17:59:53 [main] INFO  org.utplsql.cli.RunAction - ############## utPLSQL cli ##############
#                                       #
#   utPLSQL-cli 3.1.8-SNAPSHOT.local    #
#   utPLSQL-java-api 3.1.8.546          #
#   Java-Version: 1.8.0_221             #
#   ORACLE_HOME: null                   #
#   NLS_LANG: russian_russia.RU8PC866   #
#                                       #
#   Thanks for testing!                 #
#                                       #
#########################################
2019-11-26 17:59:53 [main] INFO  org.utplsql.cli.RunAction -
2019-11-26 17:59:53 [main] DEBUG o.u.c.d.TestedDataSourceProvider - NLS settings: BEGIN
EXECUTE IMMEDIATE q'[ALTER SESSION SET NLS_LANGUAGE='russian']';
EXECUTE IMMEDIATE q'[ALTER SESSION SET NLS_TERRITORY='russia']';
END;
2019-11-26 17:59:54 [main] INFO  o.u.c.d.TestedDataSourceProvider - Use connection string jdbc:oracle:thin:****/****@local_pdb
2019-11-26 17:59:54 [main] INFO  org.utplsql.cli.RunAction - Successfully connected to database. UtPLSQL core: v3.1.6.2735
2019-11-26 17:59:54 [main] INFO  org.utplsql.cli.RunAction - Oracle-Version: 12.2.0.1.0
2019-11-26 17:59:54 [main] DEBUG org.utplsql.api.reporter.Reporter - Database-reporter initialized, Type: UT_DOCUMENTATION_REPORTER, ID: 98422208E89A03F6E053020012AC69AF
2019-11-26 17:59:54 [pool-1-thread-1] INFO  org.utplsql.cli.RunTestRunnerTask - Running tests now.
2019-11-26 17:59:54 [pool-1-thread-1] INFO  org.utplsql.cli.RunTestRunnerTask - --------------------------------------
2019-11-26 17:59:54 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - TestRunner initialized
2019-11-26 17:59:54 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Running on utPLSQL v3.1.6.2735
2019-11-26 17:59:54 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Initializing reporters
2019-11-26 17:59:55 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Running tests
2019-11-26 17:59:55 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Running tests finished.
simple_payment
  sp_req_handler_pkg
    get_reqs_to_process
      ????? ????????? sp_req_handler_pkg.get_reqs_to_process
        ???? ???????? ????????????? [,004 sec]

Finished in ,007218 seconds
1 tests, 0 failed, 0 errored, 0 disabled, 0 warning(s)

2019-11-26 17:59:55 [main] INFO  org.utplsql.cli.RunAction - --------------------------------------
2019-11-26 17:59:55 [main] INFO  org.utplsql.cli.RunAction - All tests done.

Well, let's take another shot:

C:\Users\Shiryaev1-RY>set NLS_LANG=russian_cis.CL8MSWIN1251

C:\Users\Shiryaev1-RY>utplsql.bat run "some_user"/"q1w2e3r4t5y6u7i8o9p0"@local_pdb -p=SP.ut_sp_hand_get_reqs.basic_usage -d
2019-11-26 18:03:19 [main] INFO  org.utplsql.cli.RunAction - ############## utPLSQL cli ###############
#                                        #
#   utPLSQL-cli 3.1.8-SNAPSHOT.local     #
#   utPLSQL-java-api 3.1.8.546           #
#   Java-Version: 1.8.0_221              #
#   ORACLE_HOME: null                    #
#   NLS_LANG: russian_cis.CL8MSWIN1251   #
#                                        #
#   Thanks for testing!                  #
#                                        #
##########################################
2019-11-26 18:03:19 [main] INFO  org.utplsql.cli.RunAction -
2019-11-26 18:03:19 [main] DEBUG o.u.c.d.TestedDataSourceProvider - NLS settings: BEGIN
EXECUTE IMMEDIATE q'[ALTER SESSION SET NLS_LANGUAGE='russian']';
EXECUTE IMMEDIATE q'[ALTER SESSION SET NLS_TERRITORY='cis']';
END;
2019-11-26 18:03:19 [main] INFO  o.u.c.d.TestedDataSourceProvider - Use connection string jdbc:oracle:thin:****/****@local_pdb
2019-11-26 18:03:20 [main] INFO  org.utplsql.cli.RunAction - Successfully connected to database. UtPLSQL core: v3.1.6.2735
2019-11-26 18:03:20 [main] INFO  org.utplsql.cli.RunAction - Oracle-Version: 12.2.0.1.0
2019-11-26 18:03:20 [main] DEBUG org.utplsql.api.reporter.Reporter - Database-reporter initialized, Type: UT_DOCUMENTATION_REPORTER, ID: 98422E48D8890483E053020012ACCC0D
2019-11-26 18:03:20 [pool-1-thread-1] INFO  org.utplsql.cli.RunTestRunnerTask - Running tests now.
2019-11-26 18:03:20 [pool-1-thread-1] INFO  org.utplsql.cli.RunTestRunnerTask - --------------------------------------
2019-11-26 18:03:20 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - TestRunner initialized
2019-11-26 18:03:20 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Running on utPLSQL v3.1.6.2735
2019-11-26 18:03:20 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Initializing reporters
2019-11-26 18:03:20 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Running tests
2019-11-26 18:03:21 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Running tests finished.
simple_payment
  sp_req_handler_pkg
    get_reqs_to_process
      ????? ????????? sp_req_handler_pkg.get_reqs_to_process
        ???? ???????? ????????????? [,013 sec]

Finished in ,038208 seconds
1 tests, 0 failed, 0 errored, 0 disabled, 0 warning(s)

2019-11-26 18:03:21 [main] INFO  org.utplsql.cli.RunAction - --------------------------------------
2019-11-26 18:03:21 [main] INFO  org.utplsql.cli.RunAction - All tests done.

Oh, I forgot to switch chcp to 1251. And again...

C:\Users\Shiryaev1-RY>chcp 1251
Текущая кодовая страница: 1251

C:\Users\Shiryaev1-RY>utplsql.bat run "some_user"/"q1w2e3r4t5y6u7i8o9p0"@local_pdb -p=SP.ut_sp_hand_get_reqs.basic_usage -d
2019-11-26 18:03:37 [main] INFO  org.utplsql.cli.RunAction - ############## utPLSQL cli ###############
#                                        #
#   utPLSQL-cli 3.1.8-SNAPSHOT.local     #
#   utPLSQL-java-api 3.1.8.546           #
#   Java-Version: 1.8.0_221              #
#   ORACLE_HOME: null                    #
#   NLS_LANG: russian_cis.CL8MSWIN1251   #
#                                        #
#   Thanks for testing!                  #
#                                        #
##########################################
2019-11-26 18:03:37 [main] INFO  org.utplsql.cli.RunAction -
2019-11-26 18:03:37 [main] DEBUG o.u.c.d.TestedDataSourceProvider - NLS settings: BEGIN
EXECUTE IMMEDIATE q'[ALTER SESSION SET NLS_LANGUAGE='russian']';
EXECUTE IMMEDIATE q'[ALTER SESSION SET NLS_TERRITORY='cis']';
END;
2019-11-26 18:03:37 [main] INFO  o.u.c.d.TestedDataSourceProvider - Use connection string jdbc:oracle:thin:****/****@local_pdb
2019-11-26 18:03:37 [main] INFO  org.utplsql.cli.RunAction - Successfully connected to database. UtPLSQL core: v3.1.6.2735
2019-11-26 18:03:37 [main] INFO  org.utplsql.cli.RunAction - Oracle-Version: 12.2.0.1.0
2019-11-26 18:03:38 [main] DEBUG org.utplsql.api.reporter.Reporter - Database-reporter initialized, Type: UT_DOCUMENTATION_REPORTER, ID: 98422F56D51D0497E053020012AC7817
2019-11-26 18:03:38 [pool-1-thread-1] INFO  org.utplsql.cli.RunTestRunnerTask - Running tests now.
2019-11-26 18:03:38 [pool-1-thread-1] INFO  org.utplsql.cli.RunTestRunnerTask - --------------------------------------
2019-11-26 18:03:38 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - TestRunner initialized
2019-11-26 18:03:38 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Running on utPLSQL v3.1.6.2735
2019-11-26 18:03:38 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Initializing reporters
2019-11-26 18:03:38 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Running tests
2019-11-26 18:03:38 [pool-1-thread-1] INFO  org.utplsql.api.TestRunner - Running tests finished.
simple_payment
  sp_req_handler_pkg
    get_reqs_to_process
      ????? ????????? sp_req_handler_pkg.get_reqs_to_process
        ???? ???????? ????????????? [,004 sec]

Finished in ,008281 seconds
1 tests, 0 failed, 0 errored, 0 disabled, 0 warning(s)

2019-11-26 18:03:38 [main] INFO  org.utplsql.cli.RunAction - --------------------------------------
2019-11-26 18:03:38 [main] INFO  org.utplsql.cli.RunAction - All tests done.

And what about sqlplus?

C:\Users\Shiryaev1-RY>sqlplus some_user/q1w2e3r4t5y6u7i8o9p0@local_pdb

SQL*Plus: Release 11.2.0.4.0 Production on Вт Ноя 26 18:11:20 2019

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Присоединен к:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> HOST ECHO %NLS_LANG%
russian_russia.RU8PC866

SQL> @.[%NLS_LANG%].
SP2-0310: не в состоянии открыть файл ".[russian_russia.RU8PC866]..sql"
SQL> set serveroutput on
SQL> exec ut.run('SP.ut_sp_hand_get_reqs.basic_usage');
simple_payment
sp_req_handler_pkg
get_reqs_to_process
Тесты процедуры sp_req_handler_pkg.get_reqs_to_process
Тест базового использования [.004 sec]
Finished in .027698 seconds
1 tests, 0 failed, 0 errored, 0 disabled, 0 warning(s)

Процедура PL/SQL успешно завершена.

It works just fine! I call ut.run and output is okay! It's in Russian.

Hands down, I don't get how charset stuff works. Any help would be appreciated!

[pesse]

SQLDeveloper PlugIn has the same problem.

Which is sad, because it's using a different way to consume the output buffer if I remember correctly.
Any ideas, @PhilippSalvisberg ?

[pesse]

SQLcl has the same problem:

Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production


SQL> set serveroutput on
SQL> exec ut.run();
test
  ???? ???????? ????????????? [,665 sec] (FAILED - 1)

Failures:

  1) my_test_with_ru_desc
      Fail
      at "APP.RU_TEST.MY_TEST_WITH_RU_DESC", line 6 ut.fail('Fail');

Finished in ,666271 seconds
1 tests, 1 failed, 0 errored, 0 disabled, 0 warning(s)


SQL>

When trying to do a select * from table(ut.run()) we get an Encoding exception:

SQL> select * from table(ut.run());
java.io.UnsupportedEncodingException: CL8MSWIN1251
        at java.base/java.lang.StringCoding.encode(Unknown Source)
        at java.base/java.lang.String.getBytes(Unknown Source)
        at oracle.dbtools.db.ResultSetFormatter.write(ResultSetFormatter.java:710)
        at oracle.dbtools.db.ResultSetFormatter.write(ResultSetFormatter.java:703)
        at oracle.dbtools.db.SQLPLUSCmdFormatter.rset2sqlplus(SQLPLUSCmdFormatter.java:2278)
        at oracle.dbtools.db.ResultSetFormatter.rset2sqlplus(ResultSetFormatter.java:334)
        at oracle.dbtools.db.ResultSetFormatter.rset2sqlplus(ResultSetFormatter.java:308)
        at oracle.dbtools.db.ResultSetFormatter.formatResults(ResultSetFormatter.java:153)
        at oracle.dbtools.db.ResultSetFormatter.formatResults(ResultSetFormatter.java:71)
        at oracle.dbtools.raptor.newscriptrunner.SQL.processResultSet(SQL.java:798)
        at oracle.dbtools.raptor.newscriptrunner.SQL.executeQuery(SQL.java:709)
        at oracle.dbtools.raptor.newscriptrunner.SQL.run(SQL.java:83)
        at oracle.dbtools.raptor.newscriptrunner.ScriptRunner.runSQL(ScriptRunner.java:405)
        at oracle.dbtools.raptor.newscriptrunner.ScriptRunner.run(ScriptRunner.java:231)
        at oracle.dbtools.raptor.newscriptrunner.ScriptExecutor.run(ScriptExecutor.java:344)
        at oracle.dbtools.raptor.newscriptrunner.ScriptExecutor.run(ScriptExecutor.java:227)
        at oracle.dbtools.raptor.scriptrunner.cmdline.SqlCli.process(SqlCli.java:405)
        at oracle.dbtools.raptor.scriptrunner.cmdline.SqlCli.processLine(SqlCli.java:416)
        at oracle.dbtools.raptor.scriptrunner.cmdline.SqlCli.startSQLPlus(SqlCli.java:1250)
        at oracle.dbtools.raptor.scriptrunner.cmdline.SqlCli.main(SqlCli.java:492)

[PhilippSalvisberg]

@pesse where do I find the test case?

[pesse]

@PhilippSalvisberg needs a special set-up database, then I just created a testcase with some russian chars in the description:

create or replace package ru_test as
  -- %suite(test)

  -- %test(Тест базового использования)
  procedure my_test_with_ru_desc;

end;
/

create or replace package body ru_test as

  procedure my_test_with_ru_desc
  as
    begin
      ut.fail('Fail');
    end;

end;
/

[PhilippSalvisberg]

I ran it on macOS. Default client character set is there UTF-8. Looks like this:

and via SQLcl this:

[PhilippSalvisberg]

If the result of the query:

select text from user_source where name= 'RU_TEST';

looks like this

TEXT                                                                                                                                
------------------------------------------------------------------------------------------------------------------------------------
package ru_test as                                                                                                                  
  -- %suite(test)                                                                                                                   
                                                                                                                                    
  -- %test(Тест базового использования)                                                                                             
  procedure my_test_with_ru_desc;                                                                                                   
                                                                                                                                    
end;                                                                                                                                
package body ru_test as                                                                                                             
                                                                                                                                    
  procedure my_test_with_ru_desc                                                                                                    
  as                                                                                                                                
    begin                                                                                                                           
      ut.fail('Fail');                                                                                                              
    end;                                                                                                                            
                                                                                                                                    
end;                                                                                                                                

16 rows selected. 

then the NLS setup is correct and utPLSQL will show the correct test description.

In all other cases the NLS setup is wrong. It is not really a utPLSQL related issue.

[alesana-san]

Hi! In my case db encoding charset is ISO 8859-5 = CL8ISO8859P5
I guess UTF8 doesn't have these problems

[PhilippSalvisberg]

@alesana-san can you post the result of the query from SQLcl?

select text from user_source where name= 'RU_TEST';

[pesse]

Okay, I managed to get SQLcl and SQLDeveloper to output it correctly by using NLS_LANG=russian_cis.UTF8

But somehow SQLDeveloper PlugIn and cli still show ???

[PhilippSalvisberg]

@pesse where do you set NLS_LANG?

[pesse]

Some more investigation, using SQLcl (NLS_LANG set to russian_cis.UTF8):

I utPLSQL and paused it before consuming the OutputBuffer, so I could query the underlying table (user SYS):

SQL> select text from ut3.UT_OUTPUT_BUFFER_TMP;

Dez. 05, 2019 11:29:29 VORM. oracle.dbtools.raptor.newscriptrunner.ScriptExecutor run
SCHWERWIEGEND: java.base/java.lang.String.checkIndex(Unknown Source)
java.lang.StringIndexOutOfBoundsException: index 69,length 69
        at java.base/java.lang.String.checkIndex(Unknown Source)
        at java.base/java.lang.AbstractStringBuilder.charAt(Unknown Source)
        at java.base/java.lang.StringBuffer.charAt(Unknown Source)
        at oracle.dbtools.raptor.newscriptrunner.ScriptUtils.wrap(ScriptUtils.java:1134)
        at oracle.dbtools.db.SQLPLUSCmdFormatter.wrap(SQLPLUSCmdFormatter.java:2858)
        at oracle.dbtools.db.SQLPLUSCmdFormatter.rset2sqlplus(SQLPLUSCmdFormatter.java:1484)
        at oracle.dbtools.db.ResultSetFormatter.rset2sqlplus(ResultSetFormatter.java:334)
        at oracle.dbtools.db.ResultSetFormatter.rset2sqlplus(ResultSetFormatter.java:308)
        at oracle.dbtools.db.ResultSetFormatter.formatResults(ResultSetFormatter.java:153)
        at oracle.dbtools.db.ResultSetFormatter.formatResults(ResultSetFormatter.java:71)
        at oracle.dbtools.raptor.newscriptrunner.SQL.processResultSet(SQL.java:798)
        at oracle.dbtools.raptor.newscriptrunner.SQL.executeQuery(SQL.java:709)
        at oracle.dbtools.raptor.newscriptrunner.SQL.run(SQL.java:83)
        at oracle.dbtools.raptor.newscriptrunner.ScriptRunner.runSQL(ScriptRunner.java:405)
        at oracle.dbtools.raptor.newscriptrunner.ScriptRunner.run(ScriptRunner.java:231)
        at oracle.dbtools.raptor.newscriptrunner.ScriptExecutor.run(ScriptExecutor.java:344)
        at oracle.dbtools.raptor.newscriptrunner.ScriptExecutor.run(ScriptExecutor.java:227)
        at oracle.dbtools.raptor.scriptrunner.cmdline.SqlCli.process(SqlCli.java:405)
        at oracle.dbtools.raptor.scriptrunner.cmdline.SqlCli.processLine(SqlCli.java:416)
        at oracle.dbtools.raptor.scriptrunner.cmdline.SqlCli.startSQLPlus(SqlCli.java:1137)
        at oracle.dbtools.raptor.scriptrunner.cmdline.SqlCli.main(SqlCli.java:492)


TEXT
--------------------------------------------------------------------------------


  Тест базового использования 1 [,672 sec] (FAILED - 1)

Failures:

  1) my_test_with_ru_desc
      Fail
SQL> 

For me, this looks like SQLcl can output it correctly, but assumes a wrong encoding, therefore running into the index violation.

On the other hand it can query the source, containing the special chars with no problem at all:

SQL> select text from all_source where owner = 'APP';

TEXT
--------------------------------------------------------------------------------
package body ru_test as

  procedure my_test_with_ru_desc
  as
    begin
      ut.fail('Fail');
    end;
package ru_test as
  -- %suite(test)

  -- %test(Тест базового использования 1)

TEXT
--------------------------------------------------------------------------------
  procedure my_test_with_ru_desc;

end;

end;

16 Zeilen gewählt.

[pesse]

More investigation results:
Querying ut_output_buffer_tmp directly via JDBC gives the correct output. Using the Reporter/OutputBuffer does not.

[pesse]

Okay, I think I could dig it down (thanks again @PhilippSalvisberg for your help!)
There are 2 different possible issues:

Issue 1: Unpatched 12.1

I was able to reproduce a problem of different charset handling when using direct select * from... compared to using the OutputBuffer with its pipelined table functionality and SYS_REFCURSOR.
However, this problem did only occur on an unpatched 12.1 system (directly created from the official Oracle docker scripts).
It did not occur on 19.3, also not on Philipp's patched 12.1.
This is very likely an Oracle bug that is meanwhile fixed with one of the patches.

Issue 2: Console and Java File-Encoding

If I understood it correctly, Java will use the default Charset depending on the Console/Environment you use. For Windows this is mostly not UTF-8, chcp has no effect on that.
However, adding -Dfile.encoding=UTF-8 to the JVM-call does.
Unfortunately you can't just add that to the utPLSQL command (We might want to discuss if it would be beneficial to allow that).
Also, there is no way (well, there is, but a very hacky and discouraged one) to change the Default Charset once the JVM is started - which happens way before even entering the main method of any java program.
To circumvent this limitation, there's this nice little JAVA_TOOL_OPTIONS environment variable which will be read by the JVM and scanned for JVM parameters.
In my case (Powershell on Windows), setting that env variable did the trick:

> $env:JAVA_TOOL_OPTIONS="-Dfile.encoding=UTF8"
> utplsql run app/***@localhost:1523/ORCLPDB1 -d
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
2019-12-06 09:21:45 [main] INFO  org.utplsql.cli.RunAction - ######################### utPLSQL cli #########################
...

So, for NLS-Settings, use NLS_LANG environment variable.
For Charset-Settings, use JAVA_TOOL_OPTIONS environment variable with property -Dfile.encoding=YourCharset

I hope this helps and solves your problem.
I will also add this to the cli README.