• Apr 24, 2023 News!IJCTE Vol. 15, No. 1 has been indexed by SCOPUS.   [Click]
  • May 04, 2023 News!IJCTE Vol.15, No.2 has been published.   [Click]
  • Feb 08, 2023 News!IJCTE Vol. 14, No. 4 has been indexed by SCOPUS.   [Click]
General Information
    • ISSN: 1793-8201 (Print)
    • Abbreviated Title: Int. J. Comput. Theory Eng.
    • Frequency: Quarterly
    • DOI: 10.7763/IJCTE
    • Editor-in-Chief: Prof. Mehmet Sahinoglu
    • Associate Editor-in-Chief: Assoc. Prof. Alberto Arteta
    • Executive Editor: Ms. Mia Hu
    • Abstracting/Indexing: Scopus (Since 2022), INSPEC (IET), CNKI,  Google Scholar, EBSCO, etc.
    • E-mail: ijcte@iacsitp.com
    • Journal Metrics:

Prof. Mehmet Sahinoglu
Computer Science Department, Troy University, USA
I'm happy to take on the position of editor in chief of IJCTE. We encourage authors to submit papers concerning any branch of computer theory and engineering.

IJCTE 2018 Vol.10(3): 97-100 ISSN: 1793-8201
DOI: 10.7763/IJCTE.2018.V10.1206

Study of the Big Data Collection Scheme Based Apache Flume for Log Collection

Sooyong Jung and Yongtae Shin

Abstract—With the advances in IT technology and the rapid adoption of smart devices, users can more easily produce, distribute and consume data through network access anytime, anywhere. The data generated by users in response to these changes has increased dramatically. This has required companies to collect large amounts of logs, and these companies are actively researching and developing big data collection technologies. In this paper, we have studied the big data collection technology based on Apache Flume for bulk log collection. The structure for bulk log processing is designed to be matched with one web server and one Flume agent, and the Flume agents connected to the web server are connected to the Flume agent that plays the role of storing in the Hadoop distributed file system. This makes the collection of big data logs more efficient.

Index Terms—Big data, big data collection technology, Apache Flume, Apache Chukwa, hadoop distributed file system.

Sooyong Jung and Yongtae Shin are with Dept. of Computer Science Graduate School, Soongsil University, 369 Sangdo-Ro, Dongjak-Gu, Seoul, Korea (06978) (e-mail: kevinhaha777@gmail.com, sooyong.jung@gmail.com, shin@ssu.ac.kr).


Cite:Sooyong Jung and Yongtae Shin, "Study of the Big Data Collection Scheme Based Apache Flume for Log Collection," International Journal of Computer Theory and Engineering vol. 10, no. 3, pp. 97-100, 2018.

Copyright © 2008-2023. International Association of Computer Science and Information Technology. All rights reserved.