flutter - kotlin MediaProject를 이용한 실시간 내 디바이스 화면 web에 공유 (1부)

코틀린을 이용해서 실시간으로 웹에 내 디바이스 화면을 송출 하는 방법을 정리 하려고 합니다.

처음 하는 작업이라 이상한 점도 있고 어려움은 있었지만 우여곡절끝에 작업은 마무리 했습니다.

 

 

우선 MediaProject을 이용해서 사용해볼텐데요 자세한 내용은

https://developer.android.com/media/grow/media-projection?hl=ko

미디어 프로젝션  |  Android media  |  Android Developers

링크를 접속해 사용해보시길 바랍니다.

 

우선 

플러터에서 이벤트 호출시 코틀린 메소드 체널을 연결해 호출하고 서버를 통해 웹에 내 화면을 공유 하는 로직에 대해 설명하겠습니다.

 

먼저 lib/main.dart 입니다.

저의 경우는 비디오를 연결해서 실시간으로 잘보여주고있는지 확인하기 위해

video_player: ^2.7.2

 

를 이용해서 비디오를 재생 시키는 코드를 넣었습니다.

import 'package:flutter/material.dart';
import 'package:flutter/services.dart';
import 'package:video_player/video_player.dart';

void main() {
  runApp(const MaterialApp(home: ScreenSharePage()));
}

class ScreenSharePage extends StatefulWidget {
  const ScreenSharePage({super.key});

  @override
  State<ScreenSharePage> createState() => _ScreenSharePageState();
}

class _ScreenSharePageState extends State<ScreenSharePage> {
  static const platform = MethodChannel('com.example.untitled3/projection');

  late VideoPlayerController controller;
  bool isInitialized = false;

  @override
  void initState() {
    super.initState();

    controller = VideoPlayerController.networkUrl(
      Uri.parse("https://flutter.github.io/assets-for-api-docs/assets/videos/bee.mp4"),
    )..initialize().then((_) {
        controller.setLooping(true); // 반복 재생 설정
        setState(() {
          isInitialized = true;
        });
        controller.play();
      });
  }

  void _startProjection() async {
    try {
      final result = await platform.invokeMethod('startProjection');
      print("Native returned: $result");
    } catch (e) {
      print("Error  method: $e");
    }
  }

  @override
  void dispose() {
    controller.dispose();
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    final mediaQuery = MediaQuery.of(context);

    return Scaffold(
      appBar: AppBar(title: const Text("화면 공유 시작")),
      body: Container(
        height: mediaQuery.size.height,
        child: SingleChildScrollView(
          child: Column(
            children: [
              if (isInitialized)
                AspectRatio(
                  aspectRatio: controller.value.aspectRatio,
                  child: VideoPlayer(controller),
                )
              else
                const Padding(
                  padding: EdgeInsets.all(16.0),
                  child: CircularProgressIndicator(),
                ),
              const SizedBox(height: 20),
              Center(
                child: ElevatedButton(
                  onPressed: _startProjection,
                  child: const Text("Start Screen Sharing"),
                ),
              ),
            ],
          ),
        ),
      ),
    );
  }
}

 

 

이제 코틀린 쪽일텐데요

MainActivity.kt 쪽에 아래와 같이 작업합니다.

package com.example.untitled3

import android.content.Intent
import android.os.Bundle
import androidx.preference.PreferenceManager
import io.flutter.embedding.android.FlutterActivity
import io.flutter.embedding.engine.FlutterEngine
import io.flutter.plugin.common.MethodChannel

class MainActivity : FlutterActivity() {

    private val CHANNEL = "com.example.untitled3/projection"

    override fun configureFlutterEngine(flutterEngine: FlutterEngine) {
        super.configureFlutterEngine(flutterEngine)

        MethodChannel(flutterEngine.dartExecutor.binaryMessenger, CHANNEL).setMethodCallHandler { call, result ->
            if (call.method == "startProjection") {
                val prefs = PreferenceManager.getDefaultSharedPreferences(this)
                val alreadyRequested = prefs.getBoolean("projection_requested", false)

                if (!alreadyRequested) {
                    prefs.edit().putBoolean("projection_requested", true).apply()
                    val intent = Intent(this, ProjectionPermissionActivity::class.java)
                    startActivity(intent)
                    result.success("started")
                } else {
                    result.success("already_started")
                }
            } else {
                result.notImplemented()
            }
        }
    }
}

 

메소드 호출 하고 먼저 퍼미션 요청을 실행합니다.

안드로이드는 화면 녹화및 내 정보를 전송하려면 시스템 다이얼로그가 뜨고 시작하기를 눌러야 공유가 됩니다.

 

ProjectionPermissionActivity.kt

package com.example.untitled3

import android.app.Activity
import android.content.Context
import android.content.Intent
import android.media.projection.MediaProjectionManager
import android.os.Bundle
import android.util.Log
import androidx.appcompat.app.AppCompatActivity

class ProjectionPermissionActivity : AppCompatActivity() {
    private lateinit var mediaProjectionManager: MediaProjectionManager
    private val REQUEST_CODE = 1000

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        Log.d("ProjectionPermission", "화면 캡처 권한 요청 중")
        mediaProjectionManager = getSystemService(Context.MEDIA_PROJECTION_SERVICE) as MediaProjectionManager
        val permissionIntent = mediaProjectionManager.createScreenCaptureIntent()
        startActivityForResult(permissionIntent, REQUEST_CODE)
    }

    override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {
        super.onActivityResult(requestCode, resultCode, data)
        if (requestCode == REQUEST_CODE && resultCode == Activity.RESULT_OK && data != null) {
            Log.d("ProjectionPermission", "권한이 부여")

            val intent = Intent(this, ProjectionService::class.java).apply {
                putExtra("resultCode", resultCode)
                putExtra("data", data)
            }
            startService(intent)
        } else {
            Log.e("ProjectionPermission", "권한이 거부")
        }

        finish()
    }
}

 

 

이제 서비스 단에서 특정 ip로 공유를 시작합니다.

 

ProjectionService.kt

package com.example.untitled3

import android.app.*
import android.content.Context
import android.content.Intent
import android.graphics.Bitmap
import android.graphics.PixelFormat
import android.hardware.display.DisplayManager
import android.hardware.display.VirtualDisplay
import android.media.ImageReader
import android.media.projection.MediaProjection
import android.media.projection.MediaProjectionManager
import android.os.*
import android.util.Log
import okhttp3.*
import okio.ByteString
import java.io.ByteArrayOutputStream
import java.util.concurrent.TimeUnit

class ProjectionService : Service() {

    private lateinit var mediaProjectionManager: MediaProjectionManager
    private var mediaProjection: MediaProjection? = null
    private var virtualDisplay: VirtualDisplay? = null
    private var imageReader: ImageReader? = null
    private var webSocket: WebSocket? = null

    private val SERVER_URL = "ws://내IP:8080"
    private val TAG = "ProjectionService"
    private val WIDTH = 720
    private val HEIGHT = 1280

    override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
        Log.d(TAG, "onStartCommand called")

        // Foreground Service 시작
        startForegroundService()

        mediaProjectionManager = getSystemService(Context.MEDIA_PROJECTION_SERVICE) as MediaProjectionManager

        val resultCode = intent?.getIntExtra("resultCode", Activity.RESULT_CANCELED)
        val data = intent?.getParcelableExtra<Intent>("data")

        if (resultCode == null || data == null) {
            Log.e(TAG, "Missing resultCode or data")
            stopSelf()
            return START_NOT_STICKY
        }

        Log.d(TAG, "resultCode $resultCode")
        startProjection(resultCode, data)
        return START_STICKY
    }

    private fun startProjection(resultCode: Int, data: Intent) {
        Log.d(TAG, "startProjection called")
        mediaProjection = mediaProjectionManager.getMediaProjection(resultCode, data)
        imageReader = ImageReader.newInstance(WIDTH, HEIGHT, PixelFormat.RGBA_8888, 3)

        virtualDisplay = mediaProjection?.createVirtualDisplay(
            "ScreenCapture",
            WIDTH,
            HEIGHT,
            resources.displayMetrics.densityDpi,
            DisplayManager.VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR,
            imageReader?.surface,
            null,
            null
        )

        initWebSocket()
        startCaptureLoop()
    }

    private fun initWebSocket() {
        val client = OkHttpClient.Builder().readTimeout(0, TimeUnit.MILLISECONDS).build()
        val request = Request.Builder().url(SERVER_URL).build()
        webSocket = client.newWebSocket(request, object : WebSocketListener() {
            override fun onOpen(ws: WebSocket, response: Response) {
                Log.d(TAG, "WebSocket connected")
                ws.send("""{"role": "sender"}""")
            }

            override fun onFailure(ws: WebSocket, t: Throwable, response: Response?) {
                Log.e(TAG, "WebSocket failed: ${t.message}")
            }
        })
    }

    private fun startCaptureLoop() {
        val handler = Handler(Looper.getMainLooper())

        handler.post(object : Runnable {
            override fun run() {
                val image = imageReader?.acquireLatestImage()
                if (image == null) {
                    Log.w(TAG, "No image available from ImageReader")
                    handler.postDelayed(this, 100)
                    return
                }

                val planes = image.planes
                val buffer = planes[0].buffer
                val pixelStride = planes[0].pixelStride
                val rowStride = planes[0].rowStride
                val rowPadding = rowStride - pixelStride * WIDTH
                val bitmap = Bitmap.createBitmap(
                    WIDTH + rowPadding / pixelStride,
                    HEIGHT,
                    Bitmap.Config.ARGB_8888
                )
                bitmap.copyPixelsFromBuffer(buffer)
                image.close()

                Log.d(TAG, "Capturing frame")

                val stream = ByteArrayOutputStream()
                bitmap.compress(Bitmap.CompressFormat.JPEG, 50, stream)
                val jpegData = stream.toByteArray()
                Log.d(TAG, "Sending frame of size: ${jpegData.size}")
                webSocket?.send(ByteString.of(*jpegData))

                handler.postDelayed(this, 100)
            }
        })
    }

    private fun startForegroundService() {
        val channelId = "projection_channel"
        val channelName = "Screen Capture"

        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
            val channel = NotificationChannel(channelId, channelName, NotificationManager.IMPORTANCE_LOW)
            val manager = getSystemService(Context.NOTIFICATION_SERVICE) as NotificationManager
            manager.createNotificationChannel(channel)
        }

        val notification: Notification = Notification.Builder(this, channelId)
            .setContentTitle("화면 공유 중")
            .setContentText("기기의 화면을 전송 중입니다.")
            .setSmallIcon(android.R.drawable.ic_menu_camera)
            .build()

        startForeground(1, notification)
    }

    override fun onBind(intent: Intent?): IBinder? = null
}

 

테스트 환경이기 떄문에 저는 제 로컬 ip로 진행했습니다.

나중에 안될경우 방화벽도 확인해 주세요.

위에서의 작업은 Android의 MediaProjection API를 사용해 기기 화면을 캡처하고, 이를 JPEG로 압축해 WebSocket으로 전송 합니다.

 

1. onStartCommand

서비스가 시작될 때 호출
resultCode, data를 통해 MediaProjection 권한 정보 전달받음
startProjection() 호출로 화면 캡처 시작.

2. startProjection(resultCode, data)
MediaProjection 초기화
ImageReader를 생성해 화면 이미지를 실시간 캡처
VirtualDisplay를 생성해 기기 화면을 ImageReader에 출력
WebSocket 연결
캡처 루프 시작

3. initWebSocket()
OkHttp를 사용하여 WebSocket 연결을 초기화
연결되면 "role": "sender"라는 메시지를 전송
연결 실패 시 에러 로그 출력

4. startCaptureLoop()
Handler를 사용해 100ms마다 화면 이미지를 캡처
ImageReader에서 RGBA 이미지를 가져옴
Bitmap으로 변환하고 JPEG로 압축
WebSocket으로 전송

5. startForegroundService()
Android 8.0 이상에서 Foreground Service 요구사항 충족

6. onBind(...)
바인딩 안 함 → null 반환 (Service가 독립적으로 작동)

 

---

 

이제 gradle 설정을 시작합니다.

app/build.gradle

plugins {
    id "com.android.application"
    id "kotlin-android"
    // The Flutter Gradle Plugin must be applied after the Android and Kotlin Gradle plugins.
    id "dev.flutter.flutter-gradle-plugin"
}

def localProperties = new Properties()
def localPropertiesFile = rootProject.file("local.properties")
if (localPropertiesFile.exists()) {
    localPropertiesFile.withReader("UTF-8") { reader ->
        localProperties.load(reader)
    }
}

def flutterVersionCode = localProperties.getProperty("flutter.versionCode")
if (flutterVersionCode == null) {
    flutterVersionCode = "1"
}

def flutterVersionName = localProperties.getProperty("flutter.versionName")
if (flutterVersionName == null) {
    flutterVersionName = "1.0"
}

android {
    namespace = "com.example.untitled3"
    compileSdk = 34
    ndkVersion = flutter.ndkVersion

    compileOptions {
        sourceCompatibility = JavaVersion.VERSION_1_8
        targetCompatibility = JavaVersion.VERSION_1_8
    }


    defaultConfig {
        // TODO: Specify your own unique Application ID (https://developer.android.com/studio/build/application-id.html).
        applicationId = "com.example.untitled3"
        // You can update the following values to match your application needs.
        // For more information, see: https://docs.flutter.dev/deployment/android#reviewing-the-gradle-build-configuration.
        minSdk = flutter.minSdkVersion
        targetSdk = flutter.targetSdkVersion
        versionCode = flutterVersionCode.toInteger()
        versionName = flutterVersionName
    }

    buildTypes {
        release {
            // TODO: Add your own signing config for the release build.
            // Signing with the debug keys for now, so `flutter run --release` works.
            signingConfig = signingConfigs.debug
        }
    }
}

dependencies {
    implementation 'com.squareup.okhttp3:okhttp:4.9.3'
    implementation "androidx.appcompat:appcompat:1.6.1"
    implementation "androidx.preference:preference:1.2.1"
}


flutter {
    source = "../.."
}

 

build.gradle

allprojects {
    repositories {
        google()
        mavenCentral()
        maven { url 'https://maven.google.com' }
    }
}

rootProject.buildDir = "../build"
subprojects {
    project.buildDir = "${rootProject.buildDir}/${project.name}"
}

subprojects {
    project.evaluationDependsOn(":app")
}

tasks.register("clean", Delete) {
    delete rootProject.buildDir
}

 

 

AndroidManifest.xml 설정

<manifest xmlns:android="http://schemas.android.com/apk/res/android">
    <uses-permission android:name="android.permission.FOREGROUND_SERVICE_MEDIA_PROJECTION" />
    <uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
    <uses-permission android:name="android.permission.SYSTEM_ALERT_WINDOW"/>
    <uses-permission android:name="android.permission.RECORD_AUDIO" />
    <uses-permission android:name="android.permission.CAMERA" />
    <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
    <uses-permission android:name="android.permission.POST_NOTIFICATIONS"/>
    <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE"/>

    <application
        android:label="untitled3"
        android:usesCleartextTraffic="true"
        android:networkSecurityConfig="@xml/network_security_config"
        android:name="${applicationName}"
        android:icon="@mipmap/ic_launcher">

        <service
            android:name=".ProjectionService"
            android:exported="false"
            android:foregroundServiceType="mediaProjection" />

        <service
            android:name=".ScreenCaptureService"
            android:enabled="true"
            android:exported="false"
            android:foregroundServiceType="mediaProjection" />

        <activity
            android:name=".MainActivity"
            android:exported="true"
            android:launchMode="singleTop"
            android:taskAffinity=""
            android:theme="@style/LaunchTheme"
            android:configChanges="orientation|keyboardHidden|keyboard|screenSize|smallestScreenSize|locale|layoutDirection|fontScale|screenLayout|density|uiMode"
            android:hardwareAccelerated="true"
            android:windowSoftInputMode="adjustResize">
            <!-- Specifies an Android theme to apply to this Activity as soon as
                 the Android process has started. This theme is visible to the user
                 while the Flutter UI initializes. After that, this theme continues
                 to determine the Window background behind the Flutter UI. -->
            <meta-data
              android:name="io.flutter.embedding.android.NormalTheme"
              android:resource="@style/NormalTheme"
              />
            <intent-filter>
                <action android:name="android.intent.action.MAIN"/>
                <category android:name="android.intent.category.LAUNCHER"/>
            </intent-filter>
        </activity>

        <activity
            android:name=".ProjectionPermissionActivity"
            android:theme="@style/Theme.AppCompat.Light.NoActionBar"
            android:exported="false" />

        <!-- Don't delete the meta-data below.
             This is used by the Flutter tool to generate GeneratedPluginRegistrant.java -->
        <meta-data
            android:name="flutterEmbedding"
            android:value="2" />
    </application>
  
    <queries>
        <intent>
            <action android:name="android.intent.action.PROCESS_TEXT"/>
            <data android:mimeType="text/plain"/>
        </intent>
    </queries>
</manifest>

 

여기서 중요한점은 권한을 추가 해주고

각 서비스가 잘 동작할수 있도로

<service
    android:name=".ProjectionService"
    android:exported="false"
    android:foregroundServiceType="mediaProjection" />

<service
    android:name=".ScreenCaptureService"
    android:enabled="true"
    android:exported="false"
    android:foregroundServiceType="mediaProjection" />

이 내용들을 추가해주는겁니다.

 

추가로 xml 파일도 추가 해줍니다.

경로는 res/xml/network_security_config.xml

<?xml version="1.0" encoding="utf-8"?>
<network-security-config>
    <domain-config cleartextTrafficPermitted="true">
        <domain includeSubdomains="true">내 IP. 포트는 뺴주세여 (ex:192.***.***.***)</domain>
    </domain-config>
</network-security-config>

 

 

이제 모바일 쪽 에서의 잡업은 끝났습니다. 다음은 웹과 연결해 web 에서 확인할수 있게 하는 방법은 다음 글을 참고해주세요.