Replace JniType with ReferenceType; use new types in Env and caches; support dynamic class loading for proxies

[wuwbobo2021]

• Use macros for Env methods: this doesn't change any behavior.
• Simplify exception checking in Env methods: reduced some redundant code, however Env::get_class_name probably becomes a bit less strong.
• Replace JniType with ReferenceType; use new types in Env and caches: this is the major breaking change. Generated class bindings now implement ReferenceType but not JniType. Generated JNI type names for multi-dimension arrays changed from [L[I;, [L[Ljava/lang/String;; to [[I, [[Ljava/lang/String;. JClass are added for class caches. JClass, JMethodID and JFieldID are made public to be used in Env methods.
• Make proxy generation working again: this also makes it compatible with the runtime class loading "technique".

Trying to test `bluest` here

Based on akiles-dev/bluest@3c55517.

Added in java-spaghetti.yaml:

      - java/lang/ClassLoader
      - java/nio/Buffer
      - java/nio/ByteBuffer
      - dalvik/system/InMemoryDexClassLoader
      - dalvik/system/DexClassLoader
      - dalvik/system/BaseDexClassLoader

Added build.rs:

use std::env;
use std::path::PathBuf;

use android_build::{Dexer, JavaBuild};

fn main() {
    if !env::var("TARGET").unwrap().contains("android") {
        return;
    }

    let java_srcs = [
        "src/android/java/android/bluetooth/BluetoothGattCallback.java",
        "src/android/java/android/bluetooth/le/ScanCallback.java",
    ];

    let out_dir: PathBuf = env::var_os("OUT_DIR").unwrap().into();
    let out_class_dir = out_dir.join("java");

    if out_class_dir.try_exists().unwrap_or(false) {
        let _ = std::fs::remove_dir_all(&out_class_dir);
    }
    std::fs::create_dir_all(&out_class_dir)
        .unwrap_or_else(|e| panic!("Cannot create output directory {out_class_dir:?} - {e}"));

    let android_jar = android_build::android_jar(None).expect("No Android platforms found");

    // Compile the Java file into .class files
    let o = JavaBuild::new()
        .files(&java_srcs)
        .class_path(&android_jar)
        .classes_out_dir(&out_class_dir)
        .java_source_version(8)
        .java_target_version(8)
        .command()
        .unwrap_or_else(|e| panic!("Could not generate the java compiler command: {e}"))
        .args(["-encoding", "UTF-8"])
        .output()
        .unwrap_or_else(|e| panic!("Could not run the java compiler: {e}"));

    if !o.status.success() {
        panic!("Java compilation failed: {}", String::from_utf8_lossy(&o.stderr));
    }

    let o = Dexer::new()
        .android_jar(&android_jar)
        .class_path(&out_class_dir)
        .collect_classes(&out_class_dir)
        .unwrap()
        .android_min_api(20) // disable multidex for single dex file output
        .out_dir(out_dir)
        .command()
        .unwrap_or_else(|e| panic!("Could not generate the D8 command: {e}"))
        .output()
        .unwrap_or_else(|e| panic!("Error running D8: {e}"));

    if !o.status.success() {
        panic!("Dex conversion failed: {}", String::from_utf8_lossy(&o.stderr));
    }

    for java_src in java_srcs {
        println!("cargo:rerun-if-changed={java_src}");
    }
}

Changed android/adapter.rs:

use std::collections::HashMap;
use std::sync::{Arc, OnceLock};

use async_channel::Sender;
use futures_core::Stream;
use futures_lite::{stream, StreamExt};
use java_spaghetti::{ByteArray, Env, Global, Local, Null, PrimitiveArray, Ref, VM};
use tracing::{debug, warn};
use uuid::Uuid;

use super::bindings::android::bluetooth::le::{
    BluetoothLeScanner, ScanCallback, ScanResult, ScanSettings, ScanSettings_Builder,
};
use super::bindings::android::bluetooth::{BluetoothAdapter, BluetoothGattCallback, BluetoothManager};
use super::bindings::android::content::Context;
use super::bindings::android::os::ParcelUuid;
use super::bindings::java::nio;
use super::bindings::java::lang::{Class, ClassLoader, String as JString};
use super::bindings::java::util::Map_Entry;
use super::device::DeviceImpl;
use super::{JavaIterator, OptionExt};
use super::bindings::dalvik::system::InMemoryDexClassLoader;
use super::bindings::android::bluetooth::{
    BluetoothGatt, BluetoothGattCharacteristic, BluetoothGattDescriptor,
};
use crate::error::ErrorKind;
use crate::util::defer;
use crate::{
    AdapterEvent, AdvertisementData, AdvertisingDevice, ConnectionEvent, Device, DeviceId, Error, ManufacturerData,
    Result,
};

const DEX_DATA: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/classes.dex"));
static CLASS_LOADER: OnceLock<Global<ClassLoader>> = OnceLock::new();

fn init_class_loader(vm: VM, context: &Global<Context>) -> &'static Global<ClassLoader> {
    CLASS_LOADER.get_or_init(|| {
        vm.with_env(|env| {
            let context = context.as_ref(env);
            let context_loader = context.getClassLoader().unwrap();
            // Safety: casts `&[u8]` to `&[i8]`.
            let data = unsafe { std::slice::from_raw_parts(DEX_DATA.as_ptr() as *const i8, DEX_DATA.len()) };
            let byte_array = java_spaghetti::ByteArray::new_from(env, data);
            let dex_buffer = nio::ByteBuffer::wrap_byte_array(env, byte_array).unwrap();
            let dex_loader =
                InMemoryDexClassLoader::new_ByteBuffer_ClassLoader(env, dex_buffer, context_loader).unwrap();
            dex_loader.upcast::<ClassLoader>().as_global()
        })
    })
}

struct AdapterInner {
    context: Global<Context>,
    manager: Global<BluetoothManager>,
    adapter: Global<BluetoothAdapter>,
    le_scanner: Global<BluetoothLeScanner>,
}

#[derive(Clone)]
pub struct AdapterImpl {
    inner: Arc<AdapterInner>,
}

/// Creates an interface to the default Bluetooth adapter for the system.
///
/// # Safety
///
/// - The `Adapter` takes ownership of the global reference and will delete it with the `DeleteGlobalRef` JNI call when dropped. You must not do that yourself.
pub struct AdapterConfig {
    /// - `vm` must be a valid JNI `JavaVM` pointer to a VM that will stay alive for the entire duration the `Adapter` or any structs obtained from it are live.
    vm: *mut java_spaghetti::sys::JavaVM,
    /// - `context` must be a valid global reference to an `android.bluetooth.BluetoothManager` instance, from the `java_vm` VM.
    context: java_spaghetti::sys::jobject,
}

impl AdapterConfig {
    /// Creates a config for the default Bluetooth adapter for the system.
    ///
    /// # Safety
    ///
    /// - `java_vm` must be a valid JNI `JavaVM` pointer to a VM that will stay alive for the entire duration the `Adapter` or any structs obtained from it are live.
    /// - `context` must be a valid global reference to an `android.bluetooth.BluetoothManager` instance, from the `java_vm` VM.
    /// - The `Adapter` takes ownership of the global reference and will delete it with the `DeleteGlobalRef` JNI call when dropped. You must not do that yourself.
    pub unsafe fn new(java_vm: *mut java_spaghetti::sys::JavaVM, context: java_spaghetti::sys::jobject) -> Self {
        Self { vm: java_vm, context }
    }
}

impl AdapterImpl {
    /// Creates an interface to the default Bluetooth adapter for the system.
    ///
    /// # Safety
    ///
    /// In the config object:
    ///
    /// - `vm` must be a valid JNI `JavaVM` pointer to a VM that will stay alive for the entire duration the `Adapter` or any structs obtained from it are live.
    /// - `context` must be a valid global reference to an `android.bluetooth.BluetoothManager` instance, from the `java_vm` VM.
    /// - The `Adapter` takes ownership of the global reference and will delete it with the `DeleteGlobalRef` JNI call when dropped. You must not do that yourself.
    pub async fn with_config(config: AdapterConfig) -> Result<Self> {
        let vm = unsafe { VM::from_raw(config.vm) };
        let context: Global<Context> = unsafe { Global::from_raw(vm, config.context) };
        let _ = init_class_loader(vm, &context);

        vm.with_env(|env| {
            let context = context.as_local(env);
            let class: Global<Class> = unsafe {
                let jclass = env.require_class(c"android/bluetooth/BluetoothManager").unwrap();
                Global::from_raw(vm, jclass.into_raw())
            };
            let manager: Local<BluetoothManager> =
                context.getSystemService_Class(class.as_ref(env)).unwrap().unwrap().cast().unwrap();

            let adapter = manager.getAdapter()?.non_null()?;
            let le_scanner = adapter.getBluetoothLeScanner()?.non_null()?;

            Ok(Self {
                inner: Arc::new(AdapterInner {
                    context: context.as_global(),
                    adapter: adapter.as_global(),
                    le_scanner: le_scanner.as_global(),
                    manager: manager.as_global(),
                }),
            })
        })
    }

    pub(crate) async fn events(&self) -> Result<impl Stream<Item = Result<AdapterEvent>> + Send + Unpin + '_> {
        Ok(stream::empty()) // TODO
    }

    pub async fn wait_available(&self) -> Result<()> {
        Ok(())
    }

    /// Check if the adapter is available
    pub async fn is_available(&self) -> Result<bool> {
        self.inner.adapter.vm().with_env(|env| {
            let adapter = self.inner.adapter.as_local(env);
            adapter
                .isEnabled()
                .map_err(|e| Error::new(ErrorKind::Internal, None, format!("isEnabled threw: {e:?}")))
        })
    }

    pub async fn open_device(&self, id: &DeviceId) -> Result<Device> {
        self.inner.adapter.vm().with_env(|env| {
            let adapter = self.inner.adapter.as_local(env);
            let device = adapter
                .getRemoteDevice_String(JString::from_env_str(env, &id.0))
                .map_err(|e| Error::new(ErrorKind::Internal, None, format!("getRemoteDevice threw: {e:?}")))?
                .non_null()?;
            Ok(Device(DeviceImpl {
                id: id.clone(),
                device: device.as_global(),
            }))
        })
    }

    pub async fn connected_devices(&self) -> Result<Vec<Device>> {
        todo!()
    }

    pub async fn connected_devices_with_services(&self, _services: &[Uuid]) -> Result<Vec<Device>> {
        todo!()
    }

    pub async fn scan<'a>(
        &'a self,
        _services: &'a [Uuid],
    ) -> Result<impl Stream<Item = AdvertisingDevice> + Send + Unpin + 'a> {
        let (start_receiver, stream) = self.inner.manager.vm().with_env(|env| {
            let (start_sender, start_receiver) = async_channel::bounded(1);
            let (device_sender, device_receiver) = async_channel::bounded(16);

            let class_loader = CLASS_LOADER.get().unwrap().as_ref(env);
            let callback = ScanCallback::new_proxy(
                env,
                Arc::new(ScanCallbackProxy {
                    device_sender,
                    start_sender,
                }),
                class_loader.loadClass(
                    &JString::from_env_str(
                        env,
                        "com.github.alexmoon.bluest.proxy.android.bluetooth.le.ScanCallback"
                    )
                )?
                .map(|cls| java_spaghetti::JClass::from_ref(&cls).unwrap())
            )?;

            let callback_global = callback.as_global();
            let scanner = self.inner.le_scanner.as_ref(env);
            let settings = ScanSettings_Builder::new(env)?;
            settings.setScanMode(ScanSettings::SCAN_MODE_LOW_LATENCY)?;
            let settings = settings.build()?.non_null()?;
            scanner.startScan_List_ScanSettings_ScanCallback(Null, settings, callback)?;

            let guard = defer(move || {
                self.inner.manager.vm().with_env(|env| {
                    let callback = callback_global.as_ref(env);
                    let scanner = self.inner.le_scanner.as_ref(env);
                    match scanner.stopScan_ScanCallback(callback) {
                        Ok(()) => debug!("stopped scan"),
                        Err(e) => warn!("failed to stop scan: {:?}", e),
                    };
                });
            });

            Ok::<_, crate::Error>((
                start_receiver,
                Box::pin(device_receiver).map(move |x| {
                    let _guard = &guard;
                    x
                }),
            ))
        })?;

        // Wait for scan started or failed.
        match start_receiver.recv().await {
            Ok(Ok(())) => Ok(stream),
            Ok(Err(e)) => Err(e),
            Err(e) => Err(Error::new(
                ErrorKind::Internal,
                None,
                format!("receiving failed while waiting for start: {e:?}"),
            )),
        }
    }

    pub async fn discover_devices<'a>(
        &'a self,
        services: &'a [Uuid],
    ) -> Result<impl Stream<Item = Result<Device>> + Send + Unpin + 'a> {
        let connected = stream::iter(self.connected_devices_with_services(services).await?).map(Ok);

        // try_unfold is used to ensure we do not start scanning until the connected devices have been consumed
        let advertising = Box::pin(stream::try_unfold(None, |state| async {
            let mut stream = match state {
                Some(stream) => stream,
                None => self.scan(services).await?,
            };
            Ok(stream.next().await.map(|x| (x.device, Some(stream))))
        }));

        Ok(connected.chain(advertising))
    }

    pub async fn connect_device(&self, device: &Device) -> Result<()> {
        self.inner.adapter.vm().with_env(|env| {
            let device = device.0.device.as_local(env);

            let class_loader = CLASS_LOADER.get().unwrap().as_ref(env);
            let callback = BluetoothGattCallback::new_proxy(
                env,
                Arc::new(BluetoothGattCallbackProxy),
                class_loader.loadClass(
                    &JString::from_env_str(
                        env,
                        "com.github.alexmoon.bluest.proxy.android.bluetooth.BluetoothGattCallback"
                    )
                )?
                .map(|cls| java_spaghetti::JClass::from_ref(&cls).unwrap())
            )?;
            device
                .connectGatt_Context_boolean_BluetoothGattCallback(&self.inner.context, false, callback)
                .map_err(|e| Error::new(ErrorKind::Internal, None, format!("connectGatt threw: {e:?}")))?
                .non_null()?;

            Ok(())
        })
    }

// code below is unchanged ------------------------------------------------

bluest-test's Cargo.toml:

[workspace]

[package]
name = "bluest-test"
version = "0.1.0"
edition = "2024"
publish = false

[dependencies]
bluest = { path = "../bluest", features = ["unstable"] }
tracing = "0.1.36"
tracing-subscriber = "0.3.15"
# android-activity uses `log`
log = "0.4"
tracing-log = "0.2.0"
ndk-context = "0.1.1"
android-activity = { version = "0.6", features = ["native-activity"] }
futures-lite = "1.13.0"
async-channel = "2.2.0"

[lib]
crate-type = ["cdylib"]

[package.metadata.android]
package = "com.example.bluest_test"
build_targets = ["aarch64-linux-android"]

[package.metadata.android.sdk]
min_sdk_version = 23
target_sdk_version = 30

[[package.metadata.android.uses_feature]]
name = "android.hardware.bluetooth"
required = true

# TODO: support Android 12 and above, which require runtime permissions.
# <https://developer.android.google.cn/develop/connectivity/bluetooth/bt-permissions>
# <https://github.com/rust-mobile/cargo-apk/pull/72>

[[package.metadata.android.uses_permission]]
name = "android.permission.ACCESS_FINE_LOCATION"
max_sdk_version = 30

[[package.metadata.android.uses_permission]]
name = "android.permission.ACCESS_COARSE_LOCATION"
max_sdk_version = 30

[[package.metadata.android.uses_permission]]
name = "android.permission.BLUETOOTH"
max_sdk_version = 30

[[package.metadata.android.uses_permission]]
name = "android.permission.BLUETOOTH_ADMIN"
max_sdk_version = 30

bluest-test's lib.rs:

// TODO: create a simple UI with Slint.

use android_activity::{AndroidApp, MainEvent, PollEvent};
use futures_lite::{FutureExt, StreamExt};
use tracing::info;

#[unsafe(no_mangle)]
fn android_main(app: AndroidApp) {
    // android_logger::init_once(
    //     android_logger::Config::default()
    //         .with_max_level(log::LevelFilter::Info)
    //         .with_tag("bluest_test".as_bytes()),
    // );
    let subscriber = tracing_subscriber::FmtSubscriber::builder().without_time().finish();
    tracing::subscriber::set_global_default(subscriber).expect("setting tracing default failed");
    tracing_log::LogTracer::init().expect("setting log tracer failed");

    let (tx, rx) = async_channel::unbounded();
    std::thread::spawn(move || {
        let res = futures_lite::future::block_on(async_main().or(async {
            let _ = rx.recv().await;
            info!("async thread received stop signal.....");
            Ok(())
        }));
        if let Err(e) = res {
            info!("async thread's `block_on` received error {e}.");
        } else {
            info!("async thread terminates itself after it received stop signal.");
        }
    });

    let mut on_destroy = false;
    loop {
        app.poll_events(None, |event| match event {
            PollEvent::Main(MainEvent::Stop) => {
                info!("Main Stop Event.");
                let _ = tx.send(());
            }
            PollEvent::Main(MainEvent::Destroy) => {
                on_destroy = true;
            }
            _ => (),
        });
        if on_destroy {
            return;
        }
    }
}

async fn async_main() -> Result<(), Box<dyn std::error::Error>> {
    let ndk_ctx = ndk_context::android_context();
    let adapter = bluest::Adapter::with_config(unsafe {
        bluest::AdapterConfig::new(ndk_ctx.vm().cast(), ndk_ctx.context().cast())
    })
    .await?;
    adapter.wait_available().await?;

    info!("starting scan...");
    let mut scan = adapter.scan(&[]).await?;
    info!("scan started.");
    while let Some(discovered) = scan.next().await {
        info!("found a device...");
        info!("{:?}", discovered);
    }

    info!("async thread terminates itself.");
    Ok(())
}

[Dirbaio]

Why have you added the calls to register_native_method? It is not needed, Java can find them by symbol name.

[wuwbobo2021]

Code for registering native methods is added for the runtime dex class loader "technique", so that the user of some possible upstream library based on java-spaghetti doesn't have to add the needed Java code in the application's source tree (except proxy activities/services); this is essential for NativeActivity based "application"s. When a native library loads a proxy class with DexClassLoader, the linkage from the loaded class to the native library itself isn't guaranteed (I wonder if it is possible), AndroidRuntime: FATAL EXCEPTION and java.lang.UnsatisfiedLinkError will be received on the first callback.

I've added an optional argument in new_proxy for receiving the runtime-loaded class (with class name checking) if the proxy class cannot be found by Env::require_class. It would not call RegisterNatives if the proxy is found by require_class.

Also note that the object array type signature runtime generation is fixed here; this code crashes with log prompt RustPanic: couldn't load class "[L[Ljava/lang/Object;;" without the fix; correction: "[[Ljava/lang/Object;".

        let arr_obj_arr = ObjectArray::<ObjectArray<String, Throwable>, Throwable>::new(env, 10);
        let cast: &Ref<'_, ObjectArray::<ObjectArray<Object, Throwable>, Throwable>> = arr_obj_arr.cast_ref().unwrap();
        info!("Created the array.");

Side note: the current AdapterConfig in your bluest fork looks like ndk_context::AndroidContext.

[wuwbobo2021]

Changed Env::require_method and Env::require_static_method to return the possible throwable. Failure of require_class or require_field still produce panics in generated bindings.

To make unsafe fn jni_class_cache_once_lock() in added in trait ReferenceType safer (not exposing the possibility of polluting caches for classes that could have been loaded by JNI FindClass), implementations may try to initialize the lock with env.require_class(&Self::jni_reference_type_name()) before returning the reference of the lock. This will make the binding code a bit larger.

Why change ReferenceType and new_proxy for supporting runtime dex loading? The global class loader can be set for java-spaghetti, thus it can be replaced to a DexClassLoader by some java-spaghetti-based library, and the previous global class loader might be taken and used as the parent class loader when building the dex class loader; however, if there were to be multiple libraries trying to do so, the class loader hierarchy would become needlessly complicated (maybe slow).

Current logcat message of trying to create L2CAP channel on an Android 9.0 device (not crashing):

I RustStdoutStderr:  INFO bluest_test: an internal error has occured: java::lang::Throwable
I RustStdoutStderr:     getMessage:            "no non-static method \"Landroid/bluetooth/BluetoothDevice;.createInsecureL2capChannel(I)Landroid/bluetooth/BluetoothSocket;\""
...

[wuwbobo2021]

Sorry, I just found a mistake while porting the generated proxy code to the old java-spaghetti: the finalizer method isn't registered in the generated register_proxy_methods. This cannot be accepted unless I make the correction.