Skip to content

Commit 314b028

Browse files
alisonwhalisonwh
committed
Add GStreamer Neural Network Inference demo
This software demonstarates GStreamer Neural Network Inference for NXP i.MX processors. Signed-off-by: Yoshiyuki Wakinotani <[email protected]> Signed-off-by: Alison Wang <[email protected]>
0 parents  commit 314b028

31 files changed

+6246
-0
lines changed

‎COPYING

+504
Large diffs are not rendered by default.

‎Makefile.am

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# SPDX-License-Identifier: LGPL-2.0+
2+
# Copyright 2021 NXP
3+
4+
SUBDIRS = src
5+
EXTRA_DIST = autogen.sh

‎README

+81
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
GStreamer Neural Network Inference demo for NXP i.MX Processors
2+
===============================================================
3+
4+
5+
ABOUT THIS SOFTWARE
6+
-------------------
7+
This software demonstarates GStreamer Neural Network Inference for NXP i.MX processors.
8+
It uses an object detection model and a pose estimation model published in the internet.
9+
Binary distribution includes demo scripts, gstreamer plugin, TensorFlow Lite models and documents.
10+
11+
Demo scripts:
12+
/usr/bin/gstnninferencedemo-mobilenet-ssd-camera
13+
/usr/bin/gstnninferencedemo-mobilenet-ssd-video
14+
/usr/bin/gstnninferencedemo-posenet-camera
15+
/usr/bin/gstnninferencedemo-posenet-video
16+
17+
GStreamer plugin:
18+
/usr/lib/gstreamer-1.0/libgstnninferencedemo.so
19+
20+
TensorFlow Lite models and documents
21+
/usr/share/gstnninferencedemo/*
22+
23+
24+
PLATFORM
25+
--------
26+
This software is tested with below environment:
27+
1) BSP
28+
- i.MX Linux BSP 5.4.47-2.2.0
29+
2) Boards
30+
- i.MX 8M Plus EVK
31+
- i.MX 8M Mini EVK
32+
33+
34+
USAGE
35+
-----
36+
1) Install this package to imx-image-full rootfs, because it depends on OpenCV shared libs.
37+
2) Boot the i.MX EVK board and login to Linux as root
38+
3) Run below demo scripts
39+
- Object detection with video stream (720p30 video is recommended.)
40+
# gstnninferencedemo-mobilenet-ssd-video </path/to/video_file>
41+
- Object detection with camera stream
42+
# gstnninferencedemo-mobilenet-ssd-video [/dev/video?]
43+
- Pose estimation with video stream (720p30 video is recommended.)
44+
# gstnninferencedemo-posenet-video </path/to/video_file>
45+
- Pose estimation with camera stream
46+
# gstnninferencedemo-posenet-video [/dev/video?]
47+
4) If you want to play more, modify demo scripts freely!
48+
49+
50+
BUILD
51+
-----
52+
The commands to build this software in x86_64 linux host PC are below:
53+
$ tar xf gstnninferencedemo-x.y.z.tar.gz
54+
$ cd gstnninferencedemo-x.y.z
55+
$ source /opt/fsl-imx-xwayland/5.4-zeus/environment-setup-aarch64-poky-linux
56+
$ ./autogen.sh
57+
$ ./configure --host=aarch64-poky-linux
58+
$ make
59+
60+
61+
ACKNOWLEDGEMENTS
62+
----------------
63+
I'm deeply grateful to teams/communities developing great software below:
64+
65+
This software was written with reference to:
66+
- imx-gst1.0-plugin <https://source.codeaurora.org/external/imx/imx-gst1.0-plugin>
67+
68+
This software includes the works that are distributed in the Apache License 2.0.
69+
- google-coral/examples-camera <https://github.com/google-coral/examples-camera>
70+
- google-coral/project-posenet <https://github.com/google-coral/project-posenet>
71+
- google-coral/edgetpu <https://github.com/google-coral/edgetpu>
72+
73+
74+
DISCLAIMER
75+
----------
76+
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO
77+
THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT
78+
SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR
79+
ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION
80+
OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE
81+
USE OR PERFORMANCE OF THIS SOFTWARE.

‎autogen.sh

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
#!/bin/sh -ex
2+
#
3+
# SPDX-License-Identifier: LGPL-2.0+
4+
# Copyright 2021 NXP
5+
6+
# setup i.MX Linux BSP toolchain
7+
#source /opt/fsl-imx-xwayland/5.4-zeus/environment-setup-aarch64-poky-linux
8+
9+
# for m4
10+
mkdir -p m4
11+
12+
# aclocal
13+
ACLOCAL="aclocal --system-acdir=${SDKTARGETSYSROOT}/usr/share/aclocal/"
14+
15+
# aclocal path has 'automake --version'
16+
AUTOMAKE_VERSION=`automake --version | sed -e '1{s/.* //;s/\.[0-9]\+$//};q'`
17+
if [ -d ${OECORE_NATIVE_SYSROOT}/usr/share/aclocal-${AUTOMAKE_VERSION} ]; then
18+
ACLOCAL="$ACLOCAL --automake-acdir=${OECORE_NATIVE_SYSROOT}/usr/share/aclocal-${AUTOMAKE_VERSION}"
19+
fi
20+
21+
# autoreconf option
22+
ACPATHS="-I ${OECORE_NATIVE_SYSROOT}/usr/share/aclocal/"
23+
24+
# autoreconf
25+
ACLOCAL="$ACLOCAL" autoreconf -Wcross --verbose --install --force --exclude=autopoint ${ACPATHS}
26+
27+
# configure
28+
#./configure --host=aarch64-poky-linux

‎configure.ac

+137
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
# SPDX-License-Identifier: LGPL-2.0+
2+
# Copyright 2021 NXP
3+
4+
AC_PREREQ(2.59)
5+
6+
dnl initialize autoconf
7+
AC_INIT(i.MX GStreamer NN Inference demo, 0.9.1, http://www.nxp.com, gstnninferencedemo)
8+
9+
dnl initialize automake
10+
AM_INIT_AUTOMAKE([foreign tar-ustar])
11+
12+
dnl disable-static
13+
LT_INIT([disable-static])
14+
15+
dnl our libraries and install dirs use GST_API_VERSION in the filename
16+
dnl to allow side-by-side installation of different API versions
17+
GST_API_VERSION=1.0
18+
AC_SUBST(GST_API_VERSION)
19+
AC_DEFINE_UNQUOTED(GST_API_VERSION, "$GST_API_VERSION", [GStreamer API Version])
20+
21+
dnl gstreamer version requirement
22+
GST_REQ=1.6.0
23+
PKG_CHECK_MODULES(GST, gstreamer-$GST_API_VERSION >= $GST_REQ,
24+
[HAVE_GST_CHECK="yes"], [HAVE_GST_CHECK="no"
25+
AC_MSG_NOTICE([no gstreamer-$GST_API_VERSION >= $GST_REQ (GStreamer) found])
26+
])
27+
AM_CONDITIONAL(HAVE_GST_CHECK, test "x$HAVE_GST_CHECK" = "xyes")
28+
29+
dnl m4 settings
30+
AC_CONFIG_MACRO_DIR([m4])
31+
AC_SUBST(ACLOCAL_AMFLAGS, "-I m4")
32+
33+
dnl check enable feature
34+
AC_DEFUN([CHECK_ENABLE_FEATURE],
35+
[
36+
AC_ARG_ENABLE(
37+
$1,
38+
[AS_HELP_STRING([--enable-[$1]], [$2])],
39+
[enable_[$1]=$enableval],
40+
[enable_[$1]=no])
41+
if test "$enable_[$1]" = "yes"; then
42+
for h in "$4"; do
43+
AC_CHECK_HEADERS([$h], HAVE_[$3]=yes && break, HAVE_[$3]=no)
44+
done
45+
fi
46+
AM_CONDITIONAL(ENABLE_[$3], test "x$HAVE_[$3]" = "xyes")
47+
])
48+
49+
dnl AM_MAINTAINER_MODE provides the option to enable maintainer mode
50+
AM_MAINTAINER_MODE
51+
52+
dnl check for tools
53+
AC_PROG_CC
54+
AC_PROG_CXX
55+
AM_PROG_CC_C_O
56+
AC_PROG_LIBTOOL
57+
58+
dnl Check for pkgconfig first
59+
AC_CHECK_PROG(HAVE_PKGCONFIG, pkg-config, yes, no)
60+
61+
dnl Give error and exit if we don't have pkgconfig
62+
if test "x$HAVE_PKGCONFIG" = "xno"; then
63+
AC_MSG_ERROR(you need to have pkgconfig installed !)
64+
fi
65+
66+
dnl Now we're ready to ask for gstreamer libs and cflags
67+
dnl And we can also ask for the right version of gstreamer
68+
GST_REQUIRED=1.0
69+
GST_MAJORMINOR=1.0
70+
PKG_CHECK_MODULES(GST, gstreamer-$GST_MAJORMINOR >= $GST_REQUIRED, HAVE_GST=yes, HAVE_GST=no)
71+
72+
dnl Give error and exit if we don't have gstreamer
73+
if test "x$HAVE_GST" = "xno"; then
74+
AC_MSG_ERROR(you need gstreamer development packages installed !)
75+
fi
76+
77+
dnl make GST_CFLAGS and GST_LIBS available
78+
AC_SUBST(GST_CFLAGS)
79+
AC_SUBST(GST_LIBS)
80+
81+
dnl make GST_MAJORMINOR available in Makefile.am
82+
AC_SUBST(GST_MAJORMINOR)
83+
84+
dnl If we need them, we can also use the plugin libraries
85+
PKG_CHECK_MODULES(GST_BASE, gstreamer-base-$GST_MAJORMINOR >= $GST_REQUIRED, HAVE_GST_BASE=yes, HAVE_GST_BASE=no)
86+
87+
dnl Give a warning if we don't have gstreamer libs
88+
if test "x$HAVE_GST_BASE" = "xno"; then
89+
AC_MSG_ERROR(no GStreamer base found)
90+
fi
91+
92+
dnl make GST_BASE_CFLAGS and GST_BASE_LIBS available
93+
AC_SUBST(GST_BASE_CFLAGS)
94+
AC_SUBST(GST_BASE_LIBS)
95+
96+
dnl If we need them, we can also use the plugin libraries
97+
PKG_CHECK_MODULES(GST_PLUGINS_BASE, gstreamer-plugins-base-$GST_MAJORMINOR >= $GST_REQUIRED, HAVE_GST_PLUGINS_BASE=yes, HAVE_GST_PLUGINS_BASE=no)
98+
99+
dnl Give a warning if we don't have gstreamer libs
100+
if test "x$HAVE_GST_PLUGINS_BASE" = "xno"; then
101+
AC_MSG_ERROR(no GStreamer plugin base libs found)
102+
fi
103+
104+
dnl make GST_PLUGINS_BASE_CFLAGS and GST_PLUGINS_BASE_LIBS available
105+
AC_SUBST(GST_PLUGINS_BASE_CFLAGS)
106+
AC_SUBST(GST_PLUGINS_BASE_LIBS)
107+
108+
dnl add include path for imx
109+
CFLAGS="$CFLAGS -I$SDKTARGETSYSROOT/usr/include/imx"
110+
111+
dnl check vnn
112+
old_CFLAGS=$CFLAGS
113+
CFLAGS="$CFLAGS -I$SDKTARGETSYSROOT/usr/include/OVXLIB"
114+
CHECK_ENABLE_FEATURE(vnn, [internal option. don't enable.], [VNN], [vsi_nn_pub.h])
115+
CFLAGS=$old_CFLAGS
116+
117+
dnl check ion allocator header files
118+
old_CFLAGS=$CFLAGS
119+
PKG_CHECK_MODULES(GST, gstreamer-$GST_MAJORMINOR >= 1.14, CFLAGS="$CFLAGS `$PKG_CONFIG --cflags gstreamer-allocators-1.0`", CFLAGS="$CFLAGS `$PKG_CONFIG --cflags gstreamer-bad-allocators-1.0`")
120+
AC_CHECK_HEADERS([linux/ion.h gst/allocators/gstionmemory.h], HAVE_ION="yes", HAVE_ION="no")
121+
if test "x$HAVE_ION" = "xno"; then
122+
AC_MSG_ERROR(no ion allocator header found)
123+
fi
124+
AM_CONDITIONAL(USE_ION, test "x$HAVE_ION" = "xyes")
125+
CFLAGS=$old_CFLAGS
126+
127+
dnl set the plugindir where plugins should be installed
128+
plugindir="\$(libdir)/gstreamer-$GST_MAJORMINOR"
129+
AC_SUBST(plugindir)
130+
131+
dnl set proper LDFLAGS for plugins
132+
GST_PLUGIN_LDFLAGS='-module -avoid-version -export-symbols-regex gst_plugin_.*'
133+
AC_SUBST(GST_PLUGIN_LDFLAGS)
134+
135+
AC_OUTPUT(
136+
Makefile
137+
src/Makefile)

‎src/Makefile.am

+141
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
# Makefile.am for i.MX GStreamer NN Inference demo plugin
2+
#
3+
# SPDX-License-Identifier: LGPL-2.0+
4+
# Copyright 2021 NXP
5+
6+
# OpenCV
7+
#OPENCV_CXXFLAGS = `pkg-config --cflags opencv4`
8+
OPENCV_CXXFLAGS = \
9+
-std=c++11 \
10+
-I$(SDKTARGETSYSROOT)/usr/include/opencv4
11+
12+
OPENCV_LIBS = \
13+
-lopencv_highgui
14+
# -lopencv_core \
15+
# -lopencv_imgproc \
16+
# -lopencv_videoio \
17+
# -lopencv_imgcodecs
18+
19+
# Tensorflow Lite
20+
TFLITE_CXXFLAGS = \
21+
-I$(SDKTARGETSYSROOT)/usr/include/tensorflow/lite/tools/make/downloads/absl \
22+
-I$(SDKTARGETSYSROOT)/usr/include/tensorflow/lite/tools/make/downloads/flatbuffers/include
23+
24+
TFLITE_LIBS = -ltensorflow-lite
25+
26+
if ENABLE_VNN
27+
OVXLIB_CFLAGS = -I$(SDKTARGETSYSROOT)/usr/include/OVXLIB
28+
OVXLIB_LIBS = -lovxlib -lOpenVX -lOpenVXU -lCLC -lVSC -lGAL
29+
endif
30+
31+
##############################################################################
32+
# libgstnninferencedemo.la
33+
##############################################################################
34+
plugin_LTLIBRARIES = libgstnninferencedemo.la
35+
36+
HEADERS_TF = \
37+
gstnninferencedemo.h \
38+
inference.h \
39+
tflite_inference.h \
40+
tflite_benchmark.h \
41+
posenet.h \
42+
mobilenet_ssd.h \
43+
utils.h \
44+
\
45+
gstimx.h \
46+
imx_2d_device.h \
47+
imx_2d_device_allocator.h
48+
49+
HEADERS_VNN = \
50+
vnn_inference.h \
51+
vnn_global.h \
52+
vnn_tinyyolov3u8.h \
53+
vnn_tiny_yolov3.h \
54+
vnn_unetu8.h \
55+
vnn_unet.h
56+
57+
SOURCES_TF = \
58+
gstnninferencedemo.cpp \
59+
inference.cpp \
60+
tflite_inference.cpp \
61+
tflite_benchmark.cpp \
62+
posenet.cpp \
63+
mobilenet_ssd.cpp \
64+
utils.cpp \
65+
\
66+
gstimxcommon.c \
67+
imx_2d_device_allocator.c \
68+
imx_2d_device.c \
69+
imx_2d_device_g2d.c
70+
71+
SOURCES_VNN = \
72+
vnn_inference.cpp \
73+
vnn_tinyyolov3u8.c \
74+
vnn_tiny_yolov3.cpp \
75+
vnn_unetu8.c \
76+
vnn_unet.cpp
77+
78+
if ENABLE_VNN
79+
noinst_HEADERS = $(HEADERS_TF) $(HEADERS_VNN)
80+
libgstnninferencedemo_la_SOURCES = $(SOURCES_TF) $(SOURCES_VNN)
81+
else
82+
noinst_HEADERS = $(HEADERS_TF)
83+
libgstnninferencedemo_la_SOURCES = $(SOURCES_TF)
84+
endif
85+
86+
libgstnninferencedemo_la_CFLAGS = \
87+
-DUSE_G2D \
88+
$(GST_PLUGINS_BASE_CFLAGS) \
89+
$(OVXLIB_CFLAGS)
90+
91+
if USE_ION
92+
libgstnninferencedemo_la_CFLAGS += -DUSE_ION
93+
endif
94+
95+
if ENABLE_VNN
96+
libgstnninferencedemo_la_CFLAGS += -DENABLE_VNN
97+
endif
98+
99+
libgstnninferencedemo_la_CXXFLAGS = \
100+
$(libgstnninferencedemo_la_CFLAGS) \
101+
$(OPENCV_CXXFLAGS) \
102+
$(TFLITE_CXXFLAGS)
103+
104+
libgstnninferencedemo_la_LIBADD = \
105+
$(GST_PLUGINS_BASE_LIBS) \
106+
-lgstallocators-$(GST_API_VERSION) \
107+
-lgstvideo-$(GST_API_VERSION) \
108+
-lg2d \
109+
$(TFLITE_LIBS) \
110+
$(OPENCV_LIBS) \
111+
$(OVXLIB_LIBS)
112+
113+
libgstnninferencedemo_la_LDFLAGS = $(GST_PLUGIN_LDFLAGS)
114+
libgstnninferencedemo_la_LIBTOOLFLAGS = $(GST_PLUGIN_LIBTOOLFLAGS)
115+
116+
117+
# package name
118+
PACKAGE_NAME=gstnninferencedemo
119+
120+
# scripts
121+
dist_bin_SCRIPTS= \
122+
$(PACKAGE_NAME)-mobilenet-ssd-camera \
123+
$(PACKAGE_NAME)-mobilenet-ssd-video \
124+
$(PACKAGE_NAME)-posenet-camera \
125+
$(PACKAGE_NAME)-posenet-video
126+
127+
# README
128+
pkgdatadir = $(datadir)/$(PACKAGE_NAME)
129+
dist_pkgdata_DATA = \
130+
$(top_srcdir)/README \
131+
$(top_srcdir)/COPYING
132+
133+
# don't install libgstnninferencedemo.la and vnn sources
134+
install-data-hook:
135+
@(cd $(DESTDIR)/$(plugindir) && $(RM) $(plugin_LTLIBRARIES))
136+
137+
# don't distribute vnn sources if not defined ENABLE_VNN
138+
if !ENABLE_VNN
139+
dist-hook:
140+
cd $(distdir) && rm -f $(HEADERS_VNN) $(SOURCES_VNN)
141+
endif

0 commit comments

Comments
 (0)