Flutter on ARMv6

May 27, 2024

Hannes Winkler

Flutter runs on many platforms: From Android/iOS, to Web, and Desktop i.e. Windows, macOS and Linux Desktop. Those are not typically resource-constrained environments though. So, as an experiment, I thought it'd be interesting to see how flutter performs on some really resource-constrained hardware, like a Raspberry Pi Zero (1st gen)!

To put this into perspective: The Raspberry Pi Zero was released in 2015. The BCM2835 SoC that it uses is the same as the one in the original Raspberry Pi from 2012, although its single CPU core is clocked at a higher frequency of 1000MHz (vs 700MHz). The kind of CPU core that it uses, an ARM1176jzf-s, was first announced by ARM in 2003 and also used in the 1st to 3rd generation of iPhones.

Additionally, a lot of projects have already dropped support for ARMv6, the CPU architecture that the original Raspberry Pi and the Pi Zero use. Dart dropped support in 2020, flutter never actually supported it in the first place, and some of the dependencies they use rely on armv7 features.

In this blog post, I'm going to go over the modifications I had to make to the flutter engine & dart SDK, the performance on the Pi 1, and possible performance improvements you could make.

Fixing the engine

Build scripts & Toolchain

The first part of the engine that needs fixing is obviously the build scripts. The engine does kinda support configuring for arm (v7), but if you actually try to build it, it won't work. We can still try use this as a basis to try to make armv6 work, basically building our engine as an armv7 engine, but adding some compiler flags to target armv6 in reality. The patches to the build scripts are mostly to allow specifying these compiler flags. (Most importantly, --target=armv6-linux-gnueabihf, -march, -mfpu, etc.)

Generally, all the changes I made are contained in this repo: https://github.com/ardera/flutter-armv6-patches, and the build script (buildroot) changes are in the buildroot-patches directory.

The engine also comes bundled with its own pre-built clang toolchain. But when trying it I discovered it doesn't come with armv6 support anymore (it doesn't have the compiler builtins for armv6). We could try to use our distros clang, but the engine build is pretty closely fit to the specific version of clang that's have bundled, so we'd probably run into some incompatibilities.

I just opted to build my own version of the engine toolchain with armv6 support. As it turns out, that's kinda a problem on its own, LLVM doesn't really want to build with armv6 as the default target anymore (more specifically Compiler-RT), so I had to spend some time figuring out the right magic incantation of configure flags to make it compile. The exact instructions are in the Step-by-Step section at the end.

With the toolchain and build scripts in place, we can actually try to build the engine. A bit surprisingly, if we do that, the first build error we'll get is from libpng.

libpng

Apparently libpng has some optional ARM NEON (armv7 SIMD extensions) fast paths, which are unconditionally enabled when building for flutter. To fix that I just made those optimizations conditional on whether arm_use_neon is defined as a gn arg.

Dart

The next part which needed fixing was dart itself. Those fixes were a bit trickier.

As mentioned, dart removed armv6 support in 2020, so not too long ago. So at first, I tried with just reverting that removal commit and fixing some compilation issues. It seemed like that worked a bit, running a bit of dart code, but later unsurprisingly terminated with an Illegal instruction error.

Debugging revealed it was an ubfx instruction. ubfx is Unsigned Bitfield Extract , which basically takes e.g. bits 5 to 9 and stores them in some register. Similarly, there's also sbfx which sign-extends the value. It's used in several places in the dart VM:

In Assembler::ExtendValue , to move a value from one register to another and zero- or sign-extend it to a full register size at the same time.
In Assembler::ExtractClassIdFromTags to extract a class id from tags (I don't really know what that means either)

In the first occurrence, we can see that all the supported extension operations are just e.g. extending a signed byte to register size, unsigned word to register size, so no odd things like 5 bits to register size, which makes things a bit easier.

Still, armv6 is super old and resources about it are hard to find on the internet, and even then it's a bit tedious to go through the ISA manual and find a replacement by hand. So I figured I'd just get help from something that definitely knows the solution: Clang!

Fixing `Assembler::ExtendValue` using clang & godbolt

If you haven't heard of it before, godbolt is a so-called "compiler-explorer". It's an online tool that allows us to inspect the assembly output of e.g. clang, for some specific piece of code. Perfect if we want to compile some C code and see which instructions clang uses.

As I mentioned, ubfx/sbfx were used to zero/sign-extend values to register size. So let's just compile some C code that does sign- or zero-extensions with clang & godbolt, obviously with --target=armv6-linux-gnueabihf:

uint32_t extend_ub(uint32_t byte) {
    return (uint8_t) byte;
}
int32_t extend_sb(int32_t byte) {
    return (int8_t) byte;
}

uint32_t extend_uw(uint32_t word) {
    return (uint16_t) word;
}
int32_t extend_sw(int32_t word) {
    return (int16_t) word;
}

Output:

extend_ub:
        uxtb    r0, r0
        bx      lr
extend_sb:
        sxtb    r0, r0
        bx      lr

extend_uw:
        uxth    r0, r0
        bx      lr
extend_sw:
        sxth    r0, r0
        bx      lr

Perfect! Looking up these instructions, they're just more-specific versions of ubfx and sbfx that always extend a byte or word value. Since that's what we're doing in Assembler::ExtendValue anyway, we can just use those.

Fixing `Assembler::ExtractClassIdFromTags` `

The second place ubfx was used was a bit different, it extracted 20 bits from the tags register, starting from bit 12:

void Assembler::ExtractClassIdFromTags(Register result,
                                       Register tags,
                                       Condition cond) {
  ASSERT(target::UntaggedObject::kClassIdTagPos == 12);
  ASSERT(target::UntaggedObject::kClassIdTagSize == 20);
  ubfx(result, tags, target::UntaggedObject::kClassIdTagPos,
       target::UntaggedObject::kClassIdTagSize, cond);
}

At first I thought that was going to be a little trickier (well, "tricky" meaning 2 instructions). But then I saw that it's just the 20 top-most bits, so we can just do an ordinary logical right shift!

After we've patched that we can try running, and... we get another SIGILL. Looking it up, it's a dmb ish instruction this time, so a data-memory barrier. That was a bit surprising to me, dart doesn't do shared memory, so why would you need a memory barrier? Apparently, it's for isolate groups: https://github.com/dart-lang/sdk/commit/4cf584d4fed719c121c58a44df3b1cb77d5ce145

Fixing the data memory barrier

Taking the same approach again and writing some C11 atomic code in godbolt:

int load_atomic(atomic_int *a) {
    return atomic_load_explicit(a, memory_order_seq_cst);
}

We see clang emits an a weird mcr p15, #0, r1, c7, c10, #5 instruction:

load_atomic:
        mov     r1, #0
        ldr     r0, [r0]
        mcr     p15, #0, r1, c7, c10, #5
        bx      lr

Looking at the documentation, seems like that sends a zero value to a specific System Coprocessor register, which triggers data-memory barrier on armv6.

Interestingly, while implementing this, I didn't even need to code all the magic values for encoding this mcr instruction, because 11 years ago an engineer tried to use an mrc instruction for something and didn't remove the encoding values afterwards:

So, after all these patches, we can finally run flutter apps on a Pi Zero. In my case, I just used the wonders app as a test. It needs some fixes for custom embedders though, as it tries to do some unsupported things by default (setting the minimum window size using the desktop_window plugin, loading/saving via shared_preferences). The exact instructions are in the Step-by-step section again.

We have a picture!

Now, finally we can see the app running on the Pi Zero:

This content can not be viewed with respect to your privacy. Allow statistics and marketing in your or view on YouTube.

Amazingly, it actually runs relatively well. The first screen looks like it almost runs at 60fps, the second screens with the wonders pageview looks pretty good as well. Only time it seems it gets into trouble is when scrolling through the information view, with lots of widgets / effects, or the timeline view.

Stay tuned for the second part, where we'll have a closer look at the performance, e.g. look at whether it's CPU- or GPU-bottlenecked (my guess it that it's probably both), do some profiling, and try some things that could improve performance.

Step-by-step commandline instructions

1. Building the toolchain

export LLVM_CHECKOUT=~/llvm-project
export LLVM_INSTALL=~/llvm-install

apt update && apt install cmake ninja-build clang

# This would be to clone the exact commit that flutter 3.19 uses:
#  git clone -n https://llvm.googlesource.com/llvm-project.git $LLVM_CHECKOUT
#  pushd $LLVM_CHECKOUT
#  git checkout 725656bdd885483c39f482a01ea25d67acf39c46
#  popd

# However I'll use upstream clang 18.1.0 here, as that's close enough and we can
# do a shallow clone.
git clone --depth 1 -b llvmorg-18.1.0 https://github.com/llvm/llvm-project.git $LLVM_CHECKOUT

# Normally, you could install g++-arm-linux-gnueabihf and get some
# cross-compilation headers/libc, however debians `arm-linux-gnueabihf` is
# actually armv7, so we have to use our own here.
curl https://nextcloud.kdab.com/s/ncfRECJwZXgzA4d/download/armv6-linux-gnueabihf-sysroot.tar.xz | tar -xJ
export SYSROOT=$PWD/armv6-linux-gnueabihf-sysroot

cmake \
  -S $LLVM_CHECKOUT/llvm \
  -B $LLVM_CHECKOUT/build \
  -GNinja \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLVM_TARGETS_TO_BUILD=ARM \
  -DLLVM_DEFAULT_TARGET_TRIPLE=armv6-unknown-linux-gnueabihf \
  -DLLVM_ENABLE_PROJECTS="clang;lld" \
  -DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi;libunwind" \
  -DCLANG_DEFAULT_LINKER=lld \
  -DCLANG_DEFAULT_OBJCOPY=llvm-objcopy \
  -DCLANG_DEFAULT_RTLIB=compiler-rt \
  -DCLANG_DEFAULT_UNWINDLIB=libunwind \
  -DCLANG_DEFAULT_CXX_STDLIB=libc++ \
  -DLLVM_BUILTIN_TARGETS=armv6-unknown-linux-gnueabihf \
  -DBUILTINS_armv6-unknown-linux-gnueabihf_CMAKE_SYSTEM_NAME=Linux \
  -DBUILTINS_armv6-unknown-linux-gnueabihf_CMAKE_SYSROOT=$SYSROOT \
  -DBUILTINS_armv6-unknown-linux-gnueabihf_PYTHON_EXECUTABLE:PATH=$(which python) \
  -DBUILTINS_armv6-unknown-linux-gnueabihf_Python_EXECUTABLE:PATH=$(which python) \
  -DBUILTINS_armv6-unknown-linux-gnueabihf_Python3_EXECUTABLE:PATH=$(which python3) \
  -DBUILTINS_armv6-unknown-linux-gnueabihf_CMAKE_C_FLAGS="-march=armv6 -mcpu=arm1176jzf-s -mtune=arm1176jzf-s -mfpu=vfp" \
  -DBUILTINS_armv6-unknown-linux-gnueabihf_CMAKE_CXX_FLAGS="-march=armv6 -mcpu=arm1176jzf-s -mtune=arm1176jzf-s -mfpu=vfp" \
  -DBUILTINS_armv6-unknown-linux-gnueabihf_CMAKE_ASM_FLAGS="-march=armv6 -mcpu=arm1176jzf-s -mtune=arm1176jzf-s -mfpu=vfp" \
  -DLLVM_RUNTIME_TARGETS=armv6-unknown-linux-gnueabihf \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_CMAKE_SYSTEM_NAME=Linux \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_CMAKE_SYSROOT=$SYSROOT \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_CMAKE_C_FLAGS="-march=armv6 -mcpu=arm1176jzf-s -mtune=arm1176jzf-s -mfpu=vfp" \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_CMAKE_CXX_FLAGS="-march=armv6 -mcpu=arm1176jzf-s -mtune=arm1176jzf-s -mfpu=vfp" \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_CMAKE_ASM_FLAGS="-march=armv6 -mcpu=arm1176jzf-s -mtune=arm1176jzf-s -mfpu=vfp" \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_PYTHON_EXECUTABLE:PATH=$(which python) \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_Python_EXECUTABLE:PATH=$(which python) \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_Python3_EXECUTABLE:PATH=$(which python3) \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_COMPILER_RT_USE_BUILTINS_LIBRARY=OFF \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_COMPILER_RT_ENABLE_STATIC_UNWINDER=ON \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_COMPILER_RT_STATIC_CXX_LIBRARY=ON \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_COMPILER_RT_BUILD_BUILTINS=ON \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_COMPILER_RT_BUILD_LIBFUZZER=OFF \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_COMPILER_RT_BUILD_MEMPROF=OFF \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_COMPILER_RT_BUILD_PROFILE=OFF \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_COMPILER_RT_BUILD_SANITIZERS=OFF \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_COMPILER_RT_BUILD_XRAY=OFF \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_LIBUNWIND_ENABLE_SHARED=OFF \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_LIBUNWIND_USE_COMPILER_RT=ON \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_LIBCXXABI_USE_COMPILER_RT=ON \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_LIBCXXABI_ENABLE_SHARED=OFF \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_LIBCXXABI_USE_LLVM_UNWINDER=ON \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_LIBCXXABI_ENABLE_STATIC_UNWINDER=ON \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_LIBCXX_USE_COMPILER_RT=ON \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_LIBCXX_ENABLE_SHARED=OFF \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_LIBCXX_ENABLE_STATIC_ABI_LIBRARY=ON \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_LIBCXX_ENABLE_ABI_LINKER_SCRIPT=OFF \
  -DRUNTIMES_armv6-unknown-linux-gnueabihf_LIBCXX_ABI_VERSION=2 \
  -DCMAKE_INSTALL_PREFIX=$LLVM_INSTALL

ninja $LLVM_CHECKOUT/build install

2. Setting up the engine sources

apt update && apt install python-is-python3 git curl xz-utils pkg-config

# Setup depot_tools
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
export PATH="$PWD/depot_tools":"$PATH"
gclient --version  # This sets up gclient / depot_tools.

export ENGINE_ROOT=$HOME/engine

# Fetch engine (stable 3.19.6)
mkdir -p $ENGINE_ROOT && cd $ENGINE_ROOT
gclient config --spec 'solutions = [
  {
    "custom_deps": {},
    "deps_file": "DEPS",
    "managed": False,
    "name": "src/flutter",
    "safesync_url": "",
    "url": "https://github.com/flutter/engine.git",
  },
]
'
gclient sync --rev src/flutter@3.19.6

# Install sysroot for linux arm
$ENGINE_ROOT/src/build/linux/sysroot_scripts/install_sysroot.py --arch=arm

3. Patching the engine

export PATCHES=$HOME/flutter-armv6-patches
git clone https://github.com/ardera/flutter-armv6-patches $PATCHES

cd $ENGINE_ROOT/src
git am $PATCHES/buildroot-patches/*
cd $ENGINE_ROOT/src/flutter
git am $PATCHES/engine-patches/*
cd $ENGINE_ROOT/third_party/dart
git am $PATCHES/dart-patches/*
cd $ENGINE_ROOT/third_party/libpng
git am $PATCHES/libpng-patches/*

4. Configure & Build

# For whatever reason, clang tries to link against the static libraries
# in that dir instead of the shared ones with same name (e.g. libm.a instead of
# libm.so), those are not compiled with -fPIC though so linking will fail.
#
# Adding -Bdynamic, -shared etc to the compiler command-line doesn't work.
# (And at least -shared is already there anyway)
rm $SYSROOT/lib/arm-linux-gnueabihf/*.a

# Configure, tune for the Pi Zero CPU
./flutter/tools/gn \
  --embedder-for-target \
  --no-build-embedder-examples \
  --disable-desktop-embeddings \
  --no-build-glfw-shell \
  --target-os linux \
  --linux-cpu arm \
  --arm-float-abi hard \
  --runtime-mode profile \
  --no-dart-version-git-info \
  --gn-args 'verify_sdk_hash=false' \
  --target-dir 'linux_profile_armv6' \
  --target-triple armv6-linux-gnueabihf \
  --target-toolchain $LLVM_INSTALL \
  --target-sysroot $SYSROOT \
  --gn-args 'system_libdir="lib/arm-linux-gnueabihf"' \
  --gn-args 'arm_target = ""' \
  --gn-args 'arm_arch="armv6"' \
  --gn-args 'arm_cpu="arm1176jzf-s"' \
  --gn-args 'arm_tune="arm1176jzf-s"' \
  --gn-args 'arm_fpu="vfp"' \
  --gn-args 'arm_use_neon = false' \
  --gn-args 'dart_target_arch="armv6"'

# Build
ninja -C out/linux_profile_armv6

5. Building the app

# go to $HOME again
cd

# Install the flutter SDK
git clone --depth 1 -b 3.19.6 https://github.com/flutter/flutter.git
export PATH="$PATH":"$PWD/flutter/bin"
flutter precache

# Fetch the the wonders app
# This one has some patches to remove some unsupported plugins.
git clone https://github.com/ardera/flutter-wonderous-app.git wonders && cd wonders

## App build
# First, build the asset bundle. Normally there's `--local-engine`
# so we don't have to do the awkward stuff below, but for our cases this
# unfortunately doesn't work.
flutter build bundle

# Compile the app dart code for profile mode.
# Just a bunch of manual compiler invocations.
$(dirname $(which flutter))/cache/dart-sdk/bin/dartaotruntime \
  --disable-dart-dev \
  $(dirname $(which flutter))/cache/dart-sdk/bin/snapshots/frontend_server_aot.dart.snapshot \
  --aot \
  --tfa \
  --sdk-root $(dirname $(which flutter))/cache/artifacts/engine/common/flutter_patched_sdk/ \
 --target=flutter \
 --no-print-incremental-dependencies \
 -Ddart.vm.profile=true -Ddart.vm.product=false \
 --packages ./.dart_tool/package_config.json \
 --output-dill ./build/app.dill \
 --filesystem-scheme org-dartlang-root \
 --verbose \
 package:wonders/main.dart

$ENGINE_ROOT/src/out/linux_profile_armv6/clang_*/gen_snapshot \
	--deterministic \
	--snapshot_kind=app-aot-elf \
	--elf=build/flutter_assets/app.so \
	--strip \
	--sim-use-hardfp \
	./build/app.dill

2 comments on "Flutter on ARMv6"

hochmax

May 28, 2024, 9:49 AM

Impressive work

aaron.yang

Dec 10, 2024, 1:37 AM

How about ARMv7, did you know how to fix ?