Building inside a aarch64 docker fails

[mattip]

Over at MacPython/openblas-libs#169 I am trying to compile aarch64 with DYNAMIC_ARCH=1 in a manylinux2014 docker by running

docker run --platform linux/aarch64 --rm -it quay.io/pypa/manylinux2014_aarch64 /bin/bash

and then running make.

So far I am seeing these problems:

• When building without an explicit DYNAMIC_LIST, the build fails to compile the A64FX kernel:

../kernel/arm64/gemv_n_sve.c:43:18: error: ACLE function ‘svcntw’ requires ISA extension ‘sve’
    43 | #define SV_COUNT svcntw
       |                  ^~~~~~
 ../kernel/arm64/gemv_n_sve.c:62:25: note: in expansion of macro ‘SV_COUNT’
    62 |     uint64_t sve_size = SV_COUNT();
       |                         ^~~~~~~~
 ../kernel/arm64/gemv_n_sve.c:43:18: note: you can enable ‘sve’ using the command-line \
  option ‘-march’, or by using the ‘target’ attribute or pragma

since the A64FX kernel compilation does not set -march at all, and since the native architecture does not support SVE, the compilation fails.

• With the command (all the kernels except A64FX in the dynamic list)

make BUFFERSIZE=20 DYNAMIC_ARCH=1 USE_OPENMP=0 NUM_THREADS=64 BINARY=64 \
   INTERFACE64=1 SYMBOLSUFFIX=64_ LIBNAMESUFFIX=64_ 
   OBJCONV=/openblas/objconv/objconv SYMBOLPREFIX=scipy_ LIBNAMEPREFIX=scipy_ \
   FIXED_LIBNAME=1 TARGET=ARMV8 \
   DYNAMIC_LIST="ARMV8 CORTEXA53 CORTEXA57 NEOVERSEN NEOVERSEV1 NEOVERSEN2 \
                 ARMV8SVE THUNDERX THUNDERX2T99 TSV110 EMAG8180 THUNDERX3T110"

I am getting this error which echos #3189

make[1]: Entering directory `/openblas/OpenBLAS/kernel'
make[1]: warning: -jN forced in submake: disabling jobserver mode.
cc -c -O2 -DSMALL_MATRIX_OPT -DMAX_STACK_ALLOC=2048 -DDYNAMIC_LIST -DDYN_ARMV8 -DDYN_ARMV8 -DDYN_CORTEXA53 -DDYN_CORTEXA57 -DDYN_NEOVERSEN -DDYN_NEOVERSEV1 -DDYN_NEOVERSEN2 -DDYN_ARMV8SVE -DDYN_THUNDERX -DDYN_THUNDERX2T99 -DDYN_TSV110 -DDYN_EMAG8180 -DDYN_THUNDERX3T110 -Wall -DF_INTERFACE_GFORT  -fPIC -DDYNAMIC_ARCH -DSMP_SERVER -DNO_WARMUP -DBUFFERSIZE=20 -DMAX_CPU_NUMBER=64 -DMAX_PARALLEL_NUMBER=1 -DBUILD_SINGLE=1 -DBUILD_DOUBLE=1 -DBUILD_COMPLEX=1 -DBUILD_COMPLEX16=1 -DVERSION=\"0.3.27.dev\" -UASMNAME -UASMFNAME -UNAME -UCNAME -UCHAR_NAME -UCHAR_CNAME -DASMNAME=samax_k_NEOVERSEN -DASMFNAME=samax_k_NEOVERSEN_ -DNAME=samax_k_NEOVERSEN_ -DCNAME=samax_k_NEOVERSEN -DCHAR_NAME=\"samax_k_NEOVERSEN_\" -DCHAR_CNAME=\"samax_k_NEOVERSEN\" -DNO_AFFINITY -DTS=_NEOVERSEN -I.. -DBUILD_KERNEL -DTABLE_NAME=gotoblas_NEOVERSEN -UDOUBLE  -UCOMPLEX -UCOMPLEX -UDOUBLE  -DUSE_ABS -UUSE_MIN \
    ../kernel/arm64/amax.S -o samax_k_NEOVERSEN.o
make[1]: *** No rule to make target `../kernel/arm64/amin.S', needed by `samin_k_NEOVERSEN.o'.  Stop.
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory `/openblas/OpenBLAS/kernel'
make: *** [libs] Error 1

Should I be cross-compiling instead, will that work better with DYNAMIC_ARCH=1?

[mattip]

Hmm, indeed, using CC=aarch64-linux-gnu-gcc FC=aarch64-linux-gnu-gfortran HOSTCC=gcc in the make command and building outside a docker container seems to work and compile all the kernels. I will leave this open for a little while more until I work out the details of running CI with cross-compilation, but it seems compiling inside a docker is a dead-end.

[martin-frbg]

Somehow your docker environment seems to be lacking compiler support for SVE, or even a compiler version recent enough to know at least the older, non-SVE Neoverse cpu and the automatic makefile fallback to "older" -march flags appears to do more harm than good.

[mattip]

inside the docker, it is using gcc10. Does that make sense?

# gcc --version
gcc (GCC) 10.2.1 20210130 (Red Hat 10.2.1-11)

[mattip]

Ahh, right, no sve. I wonder why not:

# gcc -E -march=help -xc /dev/null
# 1 "/dev/null"
cc1: error: unknown value ‘help’ for ‘-march’
cc1: note: valid arguments are: armv8-a armv8.1-a armv8.2-a armv8.3-a armv8.4-a armv8.5-a armv8.6-a native

[martin-frbg]

10.2 should at least recognize the neoversen1 target (and Makefile.arm64 should know), so not sure why it has trouble finding an AMIN kernel for that. NeoV1 would need gcc 10.4 and sve...

[Mousius]

This is specifically mtune=a64fx, which is only available in 10.3?

Trying a fix here: #4826

[martin-frbg]

oh, the one about neoverse actually appears to be a spelling error, NEOVERSEN instead of NEOVERSEN1 and/or NEOVERSEN2 in your DYNAMIC_LIST. (There is currently no attempt made to check the contents of the list for plausibility, whoever uses this option is supposed to "know what they're doing")

[mattip]

Closing, #4826 fixes the problem when not specifying DYNAMIC_LIST. Thanks for looking into this.