re-add bison + flex tests #17

GitMensch · 2018-04-08T19:02:00Z

The first step would be to integrate the test suites back and everything that is necessary to execute them.
With bison this should be quite easy: just add the testsuite and a created atlocal script (with bison script incorporated if needed). Then all that is likely needed is a minimal MinGW/Cygwin installation + diff to execute it.

Flex seems to be a little bit more work as it doesn't use a autotest testsuite script but a self-made one based on automake. But this may also allow to run the tests with VS nmake (no unix shell interpreter needed).

With something important as the parser/tokenizer core of many systems I actually do wander why there are no tests (or did I missed something other that "the un-modified versions are tested already")...

lexxmark · 2018-04-09T05:38:57Z

With something important as the parser/tokenizer core of many systems I actually do wander why there are no tests (or did I missed something other that "the un-modified versions are tested already")...

It's all about time and efforts vs results. I only had time to refactor flex/bison code to windows environment. As windows supports linux subsystem I hope this project will be less demanded in the future. So I don't see any reason to make this project better as it's now.

But if you prefer to invest in this I think it would be valuable for the project.

Then all that is likely needed is a minimal MinGW/Cygwin installation + diff to execute it.

Is any chance to stay within VS C++ compiler?

GitMensch · 2018-04-09T10:09:13Z

As windows supports linux subsystem I hope this project will be less demanded in the future.

I don't see this. I guess you speak about the so-called "bash on windows" but this means to install some GB just to translate bison/flex sources to C.

Then all that is likely needed is a minimal MinGW/Cygwin installation + diff to execute it.

Is any chance to stay within VS C++ compiler?

Depends on what you understand as "stay within". Doing the tests from VS would mean to rewrite the complete testsuite script as VS test (the "sources" for the testsuite are available, therefore this would be possible but likely time intensive [remark: I don't know very much about the VS internal test option]).
Otherwise you need something that can execute the test script (= a shell interpreter and some standard tools like diff) or [which would be the best option but I haven't heard of any general purpose tool that works] create/use a test runner that directly executes the test definitions *.a (or an automated tool/script that change these to VS tests)).

Note: Even with a MinGW/cygwin shell executing the tests these would still be executed using the VS-generated win_flex and win_bison executables and would use cl.exe to compile their results.

lexxmark · 2018-04-10T06:33:39Z

I don't see this. I guess you speak about the so-called "bash on windows" but this means to install some GB just to translate bison/flex sources to C.

I meant Windows_Subsystem_for_Linux

Doing the tests from VS would mean to rewrite the complete testsuite script as VS test (the "sources" for the testsuite are available, therefore this would be possible but likely time intensive [remark: I don't know very much about the VS internal test option]).

I think it's not an option. As I understand Windows_Subsystem_for_Linux is pretty similar to MinGW/cygwin. So we could adopt tests to run using MinGW/cygwin and get possibility to run test under Windows_Subsystem_for_Linux for free. So user will have two options for tests.

Note: Even with a MinGW/cygwin shell executing the tests these would still be executed using the VS-generated win_flex and win_bison executables and would use cl.exe to compile their results.

I understand that.

GitMensch · 2018-04-10T07:01:35Z

I meant Windows_Subsystem_for_Linux

That is exactly what I meant - GNU/kWindows. As it is now available via Windows Store and the initial size gone down with the Debian "app" to 76MB this would be an option for people. Adding bison and flex brings in some additional MBs...

Note: Your VS build rules could be adjusted to run these tools from the WSL environment (using wsl.exe flex [...]) - Did you already tried this?

lexxmark · 2018-04-10T07:28:52Z

Note: Your VS build rules could be adjusted to run these tools from the WSL environment (using wsl.exe flex [...]) - Did you already tried this?

No I haven't tried it. I'm kinda lazy man :)
But it's a good idea.

GitMensch · 2018-09-13T14:56:12Z

If it isn't possible to add the full tests soon we should at least add an example compilation.
Thanks to @Croydon we have a working CI build now which would do the checks, too.

Actually the bug I've fixed with 6cfd835 is now many years old and would have raised a broken test in the debug environments (it likely had no actual bad result in production but could be seen directly after the commit).

I should investigate adding the original test suites with the goal to run them on CI. If this isn't possible within a reasonable time frame I go with extracting bison/flex sources and all needed files to compile them from GnuCOBOL, run bison/flex and try to use the C compiler to compile to object files. This would at least provide some general checks. @lexxmark any thoughts about adding a test subfolder containing these files?

lexxmark · 2018-09-14T05:52:26Z

Yes, please.
I would be great to have tests.

GitMensch · 2018-09-16T20:45:56Z

The first approach to just use the 2.6.4 tests with a hand-revised Makefile and testsuite runner did not work out completely (compilation of the test programs [in Release mode] is fine but the tests aren't all run) and showed some errors. Just tested using a full-configured Cygwin environment + westes/flex@4e2a81a, this time everything compiles and runs but the same test failures are in:

[...]
PASS: tableopts_opt_nr-Ca.opt.exe
PASS: tableopts_opt_nr-Ce.opt.exe
PASS: tableopts_opt_nr-Cf.opt.exe
PASS: tableopts_opt_nr-Cm.opt.exe
PASS: tableopts_opt_nr-Cem.opt.exe
PASS: tableopts_opt_nr-Cae.opt.exe
PASS: tableopts_opt_nr-Caef.opt.exe
PASS: tableopts_opt_nr-Cam.opt.exe
PASS: tableopts_opt_nr-Caem.opt.exe
PASS: tableopts_opt_r-Ca.opt.exe
PASS: tableopts_opt_r-Ce.opt.exe
PASS: tableopts_opt_r-Cf.opt.exe
PASS: tableopts_opt_r-Cm.opt.exe
PASS: tableopts_opt_r-Cem.opt.exe
PASS: tableopts_opt_r-Cae.opt.exe
PASS: tableopts_opt_r-Caef.opt.exe
PASS: tableopts_opt_r-Cam.opt.exe
PASS: tableopts_opt_r-Caem.opt.exe
FAIL: tableopts_ser_nr-Ca.ser.exe
FAIL: tableopts_ser_nr-Ca.ser.exe
FAIL: tableopts_ser_nr-Ce.ser.exe
FAIL: tableopts_ser_nr-Cf.ser.exe
FAIL: tableopts_ser_nr-Cm.ser.exe
FAIL: tableopts_ser_nr-Cem.ser.exe
FAIL: tableopts_ser_nr-Cae.ser.exe
FAIL: tableopts_ser_nr-Caef.ser.exe
FAIL: tableopts_ser_nr-Cam.ser.exe
FAIL: tableopts_ser_nr-Caem.ser.exe
FAIL: tableopts_ser_r-Ca.ser.exe
FAIL: tableopts_ser_r-Ce.ser.exe
FAIL: tableopts_ser_r-Cf.ser.exe
FAIL: tableopts_ser_r-Cm.ser.exe
FAIL: tableopts_ser_r-Cem.ser.exe
FAIL: tableopts_ser_r-Cae.ser.exe
FAIL: tableopts_ser_r-Caef.ser.exe
FAIL: tableopts_ser_r-Cam.ser.exe
FAIL: tableopts_ser_r-Caem.ser.exe
FAIL: tableopts_ver_nr-Ca.ver.exe
FAIL: tableopts_ver_nr-Ce.ver.exe
FAIL: tableopts_ver_nr-Cf.ver.exe
FAIL: tableopts_ver_nr-Cm.ver.exe
FAIL: tableopts_ver_nr-Cem.ver.exe
FAIL: tableopts_ver_nr-Cae.ver.exe
FAIL: tableopts_ver_nr-Caef.ver.exe
FAIL: tableopts_ver_nr-Cam.ver.exe
FAIL: tableopts_ver_nr-Caem.ver.exe
FAIL: tableopts_ver_r-Ca.ver.exe
FAIL: tableopts_ver_r-Ce.ver.exe
FAIL: tableopts_ver_r-Cf.ver.exe
FAIL: tableopts_ver_r-Cm.ver.exe
FAIL: tableopts_ver_r-Cem.ver.exe
FAIL: tableopts_ver_r-Cae.ver.exe
FAIL: tableopts_ver_r-Caef.ver.exe
FAIL: tableopts_ver_r-Cam.ver.exe
FAIL: tableopts_ver_r-Caem.ver.exe
[...]
# TOTAL: 113
# PASS:  69
# SKIP:  0
# XFAIL: 0
# FAIL:  44
# XPASS: 0
# ERROR: 0

test-suite.log

As cygwin64 is part of the Appveyor environment I think about just adding clwrap.sh to the repo, then after building do something like:

FLEXVER=2.6.4
wget https://github.com/westes/flex/releases/download/v$FLEXVER/flex-$FLEXVER.tar.gz
tar -xvf flex-$FLEXVER.tar.gz
cd flex-$FLEXVER
./configure
cd tests
sed -i -e 's|/dev/null|/NUL|' options.cn
make check "CC=./clwrap.sh" "CXX=./clwrap.sh -EHsc" "OBJEXT=obj" "FLEX=/cygdrive/c/projects/winflexbison/build/win_flex_bison-$APPVEYOR_REPO_TAG_NAME/win_flex.exe" "YACC=/cygdrive/c/projects/winflexbison/build/win_flex_bison-$APPVEYOR_REPO_TAG_NAME/win_bison.exe -y"

Opinions?

@lexxmark Can you please have a look at the currently failing tableopts tests (inspecting the posted test-suite.log)?

lexxmark · 2018-09-18T07:39:57Z

Opinions?

Sounds good.

Can you please have a look at the currently failing tableopts tests (inspecting the posted test-suite.log)?

I'm not familiar with clwrap.sh. Could you point me where in test-suite.log file I can get command that failed and expected result.
I will really appreciate you if you give me flex file to debug.

GitMensch · 2018-09-18T08:55:32Z

I'm not familiar with clwrap.sh

That's just a quick hack to translate the commands from the configured CC=gcc to the actual used cl.exe to use one for configuration (the normal flex environment won't configure with cl.exe) and the other one as test.

Let's inspect the test-suite.log together (I'm not familiar with the flex test-suite) and try to decipher it:

=========================

set -euvx
+ set -euvx

# testwrapper.sh: run a flex test, typically called by a Makefile

# Each test will exercise some feature or aspect of flex. Run the test with any
# input it may need.

INPUT_DIRECTORY=""
+ INPUT_DIRECTORY=
INPUT_NAME=""
+ INPUT_NAME=
INPUT_COUNT=0
+ INPUT_COUNT=0
USE_REDIRECT=0
+ USE_REDIRECT=0
DO_COMPARISON=0
+ DO_COMPARISON=0

while getopts :d:i:rt1 OPTION ; do
    case $OPTION in
        d) INPUT_DIRECTORY=$OPTARG ;;
        i)
            if [ "$INPUT_NAME" = "" ] ; then
                INPUT_NAME="$OPTARG"
            else
                INPUT_NAME="$INPUT_NAME $OPTARG"
            fi
            INPUT_COUNT=$(($INPUT_COUNT+1))
            ;;
        r) USE_REDIRECT=1 ;;
        t) USE_TABLES=1 ;;
        1) DO_COMPARISON=1 ;;
    esac
done
+ getopts :d:i:rt1 OPTION
+ case $OPTION in
+ INPUT_DIRECTORY=.
+ getopts :d:i:rt1 OPTION
+ case $OPTION in
+ '[' '' = '' ']'
+ INPUT_NAME=../../tests/tableopts.txt
+ INPUT_COUNT=1
+ getopts :d:i:rt1 OPTION
+ case $OPTION in
+ USE_REDIRECT=1
+ getopts :d:i:rt1 OPTION
+ case $OPTION in
+ USE_TABLES=1
+ getopts :d:i:rt1 OPTION

shift $(($OPTIND-1))
+ shift 6
TESTNAME=$1
+ TESTNAME=./tableopts_ser_nr-Ca.ser.exe

INPUT_NAME=${INPUT_NAME:-$INPUT_DIRECTORY/`basename "${TESTNAME%.exe}"`.txt}
+ INPUT_NAME=../../tests/tableopts.txt

if [ $DO_COMPARISON = 1 ] ; then
    TEST_OUTPUT=`$TESTNAME < $INPUT_NAME | tr -d "\r"`
    REF_OUTPUT=`$TESTNAME 1 < $INPUT_NAME | tr -d "\r"`
    test "$TEST_OUTPUT" -eq "$REF_OUTPUT"
    exit $?
fi
+ '[' 0 = 1 ']'

if [ $INPUT_COUNT -gt 1 ] ; then
    $TESTNAME ${USE_TABLES:+${INPUT_DIRECTORY}/${TESTNAME%.exe}.tables} ${INPUT_NAME}
    exit $?
fi
+ '[' 1 -gt 1 ']'

if [ -f ${INPUT_NAME} ] ; then
    if [ $USE_REDIRECT = 1 ] ; then
        $TESTNAME ${USE_TABLES:+${INPUT_DIRECTORY}/${TESTNAME%.exe}.tables} < $INPUT_NAME
    else
        $TESTNAME ${USE_TABLES:+${INPUT_DIRECTORY}/${TESTNAME%.exe}.tables} $INPUT_NAME
    fi
else
    $TESTNAME
fi
+ '[' -f ../../tests/tableopts.txt ']'
+ '[' 1 = 1 ']'
+ ./tableopts_ser_nr-Ca.ser.exe ././tableopts_ser_nr-Ca.ser.tables
table id not found in map.
FAIL tableopts_ser_nr-Ca.ser.exe (exit status: 2)

This is what I guess:

the un-prefixed parts are what the testwrapper.sh has
the parts that are effective for the specific test are marked with +

This leads us to:

TESTNAME=./tableopts_ser_nr-Ca.ser.exe
INPUT_NAME=../../tests/tableopts.txt
USE_TABLES=1
DO_COMPARISON=0 (leading to the shell script not using the comparison branch)
INPUT_COUNT=1 (leading to the shell script not using the direct call branch)
USE_REDIRECT=1
final call: ./tableopts_ser_nr-Ca.ser.exe ././tableopts_ser_nr-Ca.ser.tables
result "table id not found in map.", exit code 2

To generate tableopts_ser_nr-Ca.ser.exe we have to check the makefile.

Input: https://github.com/westes/flex/blob/master/tests/tableopts.l4
C file generated with: win_flex.exe --unsafe-no-m4-sect3-escape -P $(subst -,_,$(basename $(*F))) --tables-file="tableopts_ser_nr-Ca.ser.tables" --tables-verify $(subst _F,F,$*) -o tableopts_ser_nr-Ca.ser.c tableopts.l4
compilation done with -DTEST_HAS_TABLES_EXTERNAL (and, via clwrap.sh with -DYY_NO_UNISTD_H

This is what I've just tried on a "normal" environment with the tableopts.l4 locally available:

win_flex.exe --unsafe-no-m4-sect3-escape  --tables-file="tableopts_ser_nr-Ca.ser.tables" --tables-verify -o tableopts_ser_nr-Ca.ser.c tableopts.l4
rem change generated C file: comment config.h, replace <netinet/in.h> with <Winsock2.h>
cl.exe -DTEST_HAS_TABLES_EXTERNAL -DYY_NO_UNISTD_H tableopts_ser_nr-Ca.ser.c Ws2_32.lib
.\tableopts_ser_nr-Ca.ser.exe tableopts_ser_nr-Ca.ser.tables
   tables verification failed at flex_int8_t

I have not used generation of a tables file before this one and I have no clue how they should work, but I guess the testsuite checks that the result has an errorlevel of zero.

If something is unclear or if you need more information I'd recommend to ask in the flex mailing list / flex Github issues.

lexxmark · 2018-09-20T08:06:38Z

It seems like this issue related to windows EOL as well.
I changed EOL in win_flex and test has passed. See last commit

GitMensch · 2018-09-28T13:54:29Z

EOL is not fixed for stdout/stderr, the tests are still failing.
You can also see this with win_bison --help > help.txt where help.txt should contain unix-lf.

I assume you've missed something like

		(void)_setmode (_fileno (stdin), _O_BINARY);
		(void)_setmode (_fileno (stdout), _O_BINARY);
		(void)_setmode (_fileno (stderr), _O_BINARY);

The part that I've previously requested was to not use binary mode by default, but only if explicit requested. Otherwise you write files with \n only which may confuse some Windows editors and the stdout/stderr may occur broken on cmd.exe.
Something along the following code near the top of main() for both bison and flex:

#ifdef	_WIN32
	/* Allows running tests under woe32 */
	char *p = getenv ("USE_UNIX_LF");
	if (p && (*p == 'Y' || *p == 'y' || *p == '1')) {
		use_unix_lf = 1;  // extern int, referenced in all places to decide for "wb" or "w"
		(void)_setmode (_fileno (stdin), _O_BINARY);
		(void)_setmode (_fileno (stdout), _O_BINARY);
		(void)_setmode (_fileno (stderr), _O_BINARY);
	}
#endif

lexxmark · 2018-10-02T08:16:16Z

I have added development branch with
(void)_setmode (_fileno (stdout), _O_BINARY);
(void)_setmode (_fileno (stderr), _O_BINARY);
code added. please check

You can modify my changes or add needed code to get tests pass.
If something unclear - let me know I will try to help you.

When we finish with tests we will cleanup code (add macro guard etc) and merge it to master

lexxmark · 2019-02-27T05:10:01Z

@GitMensch have you checked development branch?
Does this branch solve EOL problems in tests?
Can we proceed further with tests adoption?

jonnysoe · 2023-03-06T07:35:37Z

Do you guys intend to use MSYS2/MinGW, or rewriting the Automake file with CTest in CMake is fine? Just that translation will be tedious as well 😅
I've mixed native MSVC with MSYS2 environment in the past, but it is more difficult to control which tool to use and I find the MinGW/GCC/Clang/UCRT bundle from WinLibs to be nicer to use

GitMensch · 2023-03-06T07:44:44Z

The original plan was to use MSYS(2), those are available on several CIs (including Appveyor, which is currently used here), because updating would then mean to just ... update, not to rewrite again.

Note: we only need the shell and maybe some autotools, the test would then run with winbison/winflex and cl.exe.
Feel free to work on this and I'll review that - but won't do the work myself any more.

jonnysoe · 2023-03-06T15:51:16Z

I see the intention is to keep test automation deviation from upstream minimal...
Just gathering requirements and understanding for now, no commitments 😉

GitMensch mentioned this issue Sep 11, 2018

Add AppVeyor CI #25

Merged

GitMensch self-assigned this Sep 13, 2018

GitMensch mentioned this issue Feb 26, 2019

Bison 3.3.2 has been released #33

Closed

GitMensch mentioned this issue Oct 23, 2020

Can not fix file via --update #62

Closed

GitMensch added Bison Issues related to Bison enhancement Flex Issues related to Flex help wanted labels Jun 22, 2021

GitMensch mentioned this issue Jan 24, 2022

Implemented compatibility to MinGW-w64 #59

Open

GitMensch removed their assignment Mar 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

re-add bison + flex tests #17

re-add bison + flex tests #17

GitMensch commented Apr 8, 2018

lexxmark commented Apr 9, 2018

GitMensch commented Apr 9, 2018 •

edited

Loading

lexxmark commented Apr 10, 2018

GitMensch commented Apr 10, 2018

lexxmark commented Apr 10, 2018

GitMensch commented Sep 13, 2018

lexxmark commented Sep 14, 2018

GitMensch commented Sep 16, 2018

lexxmark commented Sep 18, 2018

GitMensch commented Sep 18, 2018

lexxmark commented Sep 20, 2018

GitMensch commented Sep 28, 2018

lexxmark commented Oct 2, 2018

lexxmark commented Feb 27, 2019

jonnysoe commented Mar 6, 2023

GitMensch commented Mar 6, 2023

jonnysoe commented Mar 6, 2023

re-add bison + flex tests #17

re-add bison + flex tests #17

Comments

GitMensch commented Apr 8, 2018

lexxmark commented Apr 9, 2018

GitMensch commented Apr 9, 2018 • edited Loading

lexxmark commented Apr 10, 2018

GitMensch commented Apr 10, 2018

lexxmark commented Apr 10, 2018

GitMensch commented Sep 13, 2018

lexxmark commented Sep 14, 2018

GitMensch commented Sep 16, 2018

lexxmark commented Sep 18, 2018

GitMensch commented Sep 18, 2018

lexxmark commented Sep 20, 2018

GitMensch commented Sep 28, 2018

lexxmark commented Oct 2, 2018

lexxmark commented Feb 27, 2019

jonnysoe commented Mar 6, 2023

GitMensch commented Mar 6, 2023

jonnysoe commented Mar 6, 2023

GitMensch commented Apr 9, 2018 •

edited

Loading