Accelerated MIPI CSI video stream acquision in tasks of real-time video streaming
DOI:
https://doi.org/10.20535/RADAP.2020.82.35-43Keywords:
SoC, FPGA, MIPI CSI, GigE Vision, video, streamingAbstract
In present study the challenges of reducing transmission latencies of a real-time video stream acquired from cameras connected via the MIPI CSI interface were addressed. In the study, the main components of the video stream acquisition/transmission latency are given, the degree of their contribution to the total latency was analyzed, the assessment on the potential ability to influence them when developing a real-time video stream acquisition/transmission systems was given. The issues connected with using the frame buffering in such systems are designated, primarily the impact on the total latency value when having a framebuffer in such a system. The limitations of the existing MIPI module implementations of some ARM microprocessors resulting in latency increase for MIPI CSI camera video stream acquisition were characterized.
The structural and functional organization based on the use of digital streaming buses, fragmentation of video frames and DMA transactions for MIPI CSI video stream acquisition systems was proposed, which does not require the use of framebuffers and, as a result, provides the possibility of reducing the overall video stream acquisition latency. The proposed structural and functional organization could be implemented based on SoC-FPGA solutions, including the use of the existing IP-cores. Pragmatic peculiar features were described and the corresponding expression for estimating the limiting value of the latency for the proposed structural and functional organization was given.
For experimental verification, a prototype of the video stream acquisition/transmission system, based on the Zynq-7000 SoC-FPGA family of Xilinx following the proposed structural and functional organization was created. Its specifics and corresponding features of its implementation were discussed in the paper. The performance of the obtained prototype was estimated, and the possible directions towards further reduction of the overall latency of video stream acquisition/transmission were considered.
The results of the study may prove useful to reduce the latencies of the video streams acquired from MIPI CSI cameras in real-time video stream transmission systems based on SoC-FPGA.
References
Перелік посилань
MIPI Camera Serial Interface 3 (MIPI CSI-3)/MIPI Alliance
Ходнев Т. А. Оценка эффективности использования тракта связи протоколами RTSP-видеовещания в задачах передачи видеопоследовательностей реального времени / Т. А. Ходнев, А. Ю. Варфоломеев // XІІ Міжнародно науково-технічна конференція ``Проблеми телекомунікацій'' ПТ-2018: Збірник матеріалів конференції. — Київ : [КПІ ім. Ігоря Сікорського], 2018. — 500 с. — с. 332-335.
Khodniev T. A., Varfolomieiev A. Y., Lysenko O. M. , Antonyuk O. I. (2018) Comparison of RTSP and,GigE Vision video streaming technologies in terms of communication path utilization efficiency: an experimental approach. 2018 International Conference on Information and Telecommunication Technologies and Radio Electronics (UkrMiCo). Одеса, с. 1-4. DOI:10.1109/UkrMiCo43733.2018.9047531.
Марченко В. І. Програмно-апаратна реалізація відеокамери, сумісної зі стандартом GigE Vision / В. І. Марченко, Т. А. Ходнєв, А. Ю. Варфоломеєв // Мікросистеми, Електроніка та Акустика. — 2018. — T. 23, №,5. — с. 32-37. DOI : 10.20535/2523-4455.2018.23.5.147686.
Ходнев Т. А. Поуровнево-декомпозиционная модель оценки интегральной эффективности использования тракта связи с учетом помех / Т. А. Ходнев, А. И. Антонюк, А. Ю. Варфоломеев, А. Н. Лысенко // Мікросистеми, Електроніка та Акустика. — 2018. — T. 23, № 6. — с. 29-33. DOI : 10.20535/2523-4455.2018.23.6.154720.
Rolling Shutter vs. Global Shutter / Teledyne QImaging
Liang C. K. Analysis and compensation of rolling shutter effect / C. K. Liang, L. W. Chang, H. H. Chen // IEEE Transactions on Image Processing. — 2008. — T. 17, № 8. — с. 1323-1330. DOI : 10.1109/TIP.2008.925384
OV5640: color CMOS QSXGA (5 megapixel) image sensor with OmniBSI technology / OmniVision Technologies Inc.
Low-latency design considerations for video-enabled drones (SPRY301) / Texas Instruments Inc.
Ahmad J. FPGA based Deterministic Latency Image Acquisition and Processing System for Automated Driving Systems / J. Ahmad, A. Warren // 2018 IEEE International Symposium on Circuits and Systems (ISCAS). — 2018. DOI : 10.1109/iscas.2018.8351472.
Understanding and Reducing Latency in Video Compression Systems / CAST Inc.
MIPI-CSI2 Peripheral on i.MX6 MPUs (AN5305) / NXP Semiconductors Inc.
Camera Abstraction Layer - Processor SDK Linux Documentation / Texas Instruments Inc.
AMBA 4 AXI4-Stream Protocol / ARM Ltd.
AXI4-Stream Infrastructure IP Suite v3.0 (PG085) / Xilinx Inc.
Jayakrishnan V. Embedded Processors on FPGA: Soft vs Hard / V. Jayakrishnan, C. Parikh // Proceedings of the 2019 ASEE North Central Section Conference. — 2019
ARM A-Profile Architecture Specifications / ARM Ltd.
Zynq-7000 SoC Data Sheet: Overview (DS190) / Xilinx Inc.
Siewert S. Real-Time Embedded Components And Systems Using Linux And RTOS / Siewert S. and Pratt J. — 2-ге вид. — Dulles, VA : Mercury Learning and Information, 2016. — 500 c. — ISBN: 1942270046
Digilent Design Contest 2019 EU Region Finalists / Digilent Inc.
Pcam 5C: 5 MP Color Camera Sensor / Digilent Inc.
Zybo Z7: Zynq-7000 ARM/FPGA SoC Development Board / Digilent Inc.
ARAVIS: A vision library for genicam based cameras / Github
Vivado Design Suite / Xilinx Inc.
MIPI CSI-2 Receiver Subsystem v4.1 IP (PG232) / Xilinx Inc.
PetaLinux Tools / Xilinx Inc.
Yocto Project / Linux Foundation
Meta-aravis: Yocto layer for the Aravis application / Github
V4L2 API Input/Output: Streaming I/O (DMA buffer importing) / Linux Kernel Organization Inc.
Т. А. Ходнєв, М. С. Голуб, О. В. Кужильний. Gigevision-xilinx: GigE Vision compatibe video streaming from MIPI-CSI camera with Zybo Z7-10 board
AXI Video Direct Memory Access v6.3 (PG020) /
Video Frame Buffer Read and Video Frame Buffer Write v2.1 (PG278) /
References
MIPI Alliance, MIPI Camera Serial Interface 3 (MIPI CSI-3), standard, viewed 18 Jun 2020.
Khodniev T. A. and Varfolomieiev A. Y. (2018) Evaluating the efficiency of communication path utilization by RTSP broadcasting protocols in tasks of real-time video sequences transmission. Conference materials of XIIth International Scientific Conference ``Modern challenges in telecommunications'', pp. 332-335, in Russian.
Khodniev T. A., Varfolomieiev A. Y., Lysenko O. M. and Antonyuk O. I. (2018) Comparison of RTSP and GigE Vision video streaming technologies in terms of communication path utilization efficiency: an experimental approach. 2018 International Conference on Information and Telecommunication Technologies and Radio Electronics (UkrMiCo), pp.1-4. DOI: 10.1109/ukrmico43733.2018.9047531.
Marchenko V. I., Khodniev T. A. and Varfolomieiev A. Y. (2018) Software and Hardware Implementation of Video Camera, Compatible with GigE Vision Standard. Microsystems, Electronics and Acoustics, Vol. 23, Iss. 5, pp. 32-37. DOI: 10.20535/2523-4455.2018.23.5.147686.
Khodniev T. A., Antoniuk O. I., Varfolomieiev A. Y. and Lysenko O. M. (2018) By-Layer Decomposition Model for Evaluating the Integral Communication Path Utilization Efficiency with Account for Errors. Microsystems, Electronics and Acoustics, Vol. 23, Iss. 6, pp. 29-33. DOI: 10.20535/2523-4455.2018.23.6.154720.
Teledyne QImaging, Rolling Shutter vs. Global Shutter, technical note, viewed 18 Jun 2020.
Liang C., Chang L. and Chen H. (2008) Analysis and Compensation of Rolling Shutter Effect. IEEE Transactions on Image Processing, Vol. 17, Iss. 8, pp. 1323-1330. DOI: 10.1109/tip.2008.925384.
OV5640: color CMOS QSXGA (5 megapixel) image sensor with OmniBSI technology. OmniVision Technologies Inc., datasheet, archived, viewed 18 Jun 2020.
Low-latency design considerations for video-enabled drones (SPRY301). Texas Instruments Inc., application note, viewed 18 Jun 2020.
Ahmad J. and Warren A. (2018) FPGA based Deterministic Latency Image Acquisition and Processing System for Automated Driving Systems. 2018 IEEE International Symposium on Circuits and Systems (ISCAS). DOI: 10.1109/iscas.2018.8351472
Understanding and Reducing Latency in Video Compression Systems. CAST Inc., viewed 18 Jun 2020.
MIPI-CSI2 Peripheral on i.MX6 MPUs (AN5305). NXP Semiconductors Inc., application note, viewed 18 Jun 2020.
Camera Abstraction Layer - Processor SDK Linux Documentation. Texas Instruments Inc., developer's guide, viewed 18 Jun 2020.
AMBA 4 AXI4-Stream Protocol. ARM Ltd., specification, viewed 18 Jun 2020.
AXI4-Stream Infrastructure IP Suite v3.0 (PG085). Xilinx Inc., product guide, viewed 18 Jun 2020.
Jayakrishnan V. and Parikh C. (2019) Embedded Processors on FPGA: Soft vs Hard. Proceedings of the 2019 ASEE North Central Section Conference.
ARM A-Profile Architecture Specifications. ARM Ltd., reference manual, viewed 18 Jun 2020.
Zynq-7000 SoC Data Sheet: Overview (DS190). Xilinx Inc., product specification, viewed 18 Jun 2020.
Siewert S. and Pratt J. (2016) textit{Real-Time Embedded Components And Systems Using Linux And RTOS}, 2nd ed., Dulles, Virginia: Mercury Learning and Information, ISBN: 1942270046
Digilent Design Contest 2019 EU Region Finalists. Digilent Inc.
Pcam 5C: 5 MP Color Camera Sensor. Digilent Inc., product page, viewed 18 Jun 2020.
Zybo Z7: Zynq-7000 ARM/FPGA SoC Development Board. Digilent Inc., product page, viewed 18 Jun 2020.
ARAVIS: A vision library for genicam based cameras. GitHub, software library repository, viewed 18 Jun 2020.
Vivado Design Suite. Xilinx Inc., product page, viewed 18 Jun 2020.
MIPI CSI-2 Receiver Subsystem v4.1 IP (PG232). Xilinx Inc., product guide, viewed 18 Jun 2020.
PetaLinux Tools. Xilinx Inc., product page, viewed 18 Jun 2020.
Yocto Project, Linux Foundation project home, viewed 18 Jun 2020.
Meta-aravis: Yocto layer for the Aravis application. GitHub, software repository, viewed 18 Jun 2020.
V4L2 API Input/Output: Streaming I/O (DMA buffer importing). Linux Kernel Organization Inc., Linux kernel v. 4.9 documentation, viewed 18 Jun 2020.
Khodniev T. A., Holub M. S. and Kuzhylnyi O. V. (2020) Gigevision-xilinx: GigE Vision compatibe video streaming from MIPI-CSI camera with Zybo Z7-10 board. GitHub, project software repository, viewed 18 Jun 2020.
AXI Video Direct Memory Access v6.3 (PG020). Xilinx Inc., product guide, viewed 18 Jun 2020.
Video Frame Buffer Read and Video Frame Buffer Write v2.1 (PG278). Xilinx Inc., product guide, viewed 18 Jun 2020.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2020 Т. А. Ходнєв , М. C. Голуб, О. В. Кужильний, О. М. Лисенко, А. Ю. Варфоломєєв
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
